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ABSTRACT 



This report summarizes the proceedings of an Office of 
Educational Research and Improvement (OERI) sponsored meeting. The discussion 
and followup papers address issues related to disseminating promising and 
exemplary programs in conjunction with OERI ' s responsibilities to plan a 
system of expert panels . Section 1 of the report includes the minutes of the 
planning meeting, a summary of the themes from the discussion groups, the 
agenda, a list of participants, and meeting handouts. Section 2, making up 
more than half the report, contains the individual commissioned papers. They 
are: "An Overview of OERI Efforts to Develop a System to Designate and 
Disseminate Promising and Exemplary Products, Programs and Practices" (Susan 
Klein) ; "Expert Panels and the Dissemination of Exemplary Practices" (John 
Evans); "The System of Expert Panels: Reflections on the Future from Lessons 
of the Past Twenty Years" (M. Christine Dwyer); "Sharing the Best We've Got 
to Offer: Reflections on a System of Expert Panels to Designate and 
Disseminate Promising and Exemplary Products, Programs, and Practices (John 
Luczak) ; "Dissemination and Evaluation Issues Related to the System of Expert 
Panels" (Gary Borich) ; "The Expert Panels to be Charged with Identifying 
Promising and Exemplary Educational Programs: Issues and Suggestions" (Janet 
Carter); "Other Key Issues for Planning a System of Expert Panels: Making a 
Complex System Viable, Inclusive, and Fair" (Floraline Stevens) ; "Expert 
Panels and Educational Reforms: Some Perceptions Based on a Search for 
Promising and Exemplary Adolescent Pregnancy Programs" (Karen Bogart) ; 
"Reactions to the Expert Panel System as Piloted by OERI" (Lynn Fox) ; "Some 
Thoughts and Reflections on the OERI Expert Panel System" (Harilyn Rousso) ; 
"The Gender Equity Expert Panel: The Core Gender Equity Sub-Panel One Year 
Later" (Marylin A. Hulme) ; "Advice for Planning a System of Expert Panels" . 
(Mary M. Wiberg) ; and "Lessons to be Learned from Setting Up the Gender 
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Preface 



This report on an Office of Educational Research and Improvement (OERI) sponsored meeting and 
follow-up papers addresses issues related to disseminating promising and exemplary programs in 
conjunction with OERI’s responsibilities to plan a system of expert panels. The meeting was held 
the morning on March 24, 1997 in Chicago just before the 1997 meeting of the American 
Educational Research Association. Selected invited participants were each asked to prepare a paper 
due after the meeting to discuss additional insights on the development of this system of expert 
panels. Some of the attendees were also presenters in regular AERA sessions on the afternoon on 
March 24 on “Learning from Consumer-Oriented Review Efforts to Designate Promising and 
Exemplary Products, Programs and Practices” and the March 25 session on “The Gender Equity 
Expert Panel.” 

Section I of this report includes the morning planning meeting minutes, a summary of themes from 
the discussion groups, the agenda, list of participants, and meeting handouts. 

Section II contains the individual commissioned papers. These papers are organized to provide: 

• an overview of the evolving system of expert panels (Paper A) 

• previous federal education efforts to identify and share research-based promising and 

exemplary products, programs and practices (Papers B&C) 

• review efforts of other agencies, foundations and education sponsors (Papers D-H) 

• insights gained from experiences pilot testing the gender equity expert panel (Papers I-M) 

In addition to thanking all the participants and the authors, we would like to thank DTI Associates 
of Arlington, VA for their fine staff work in facilitating this meeting and in preparing this report. 

The Knowledge Applications Division in the Office of Reform Assistance and Dissemination 
(ORAD) in OERI which has lead responsibility for developing this System of Expert Panels will be 
issuing related reports and commissioned papers. Many of these are referenced in one of the meeting 
handouts at the end of Section I and are now completed. In addition to continued work of the pilot 
expert panels in Mathematics and Science Education and Gender Equity, there is a 1 997 contract to 
support the work of new pilot expert panels in Early Reading, Technology and Safe and Drug Free 
Schools. 

If you are interested in keeping informed about the evolving system of expert panels feel free to 
contact us. If you would like to participate in the “findbest” listserv on expert panel activities, please 
e-mail Tara_Ariola@ed.gov and send her your e-mail address and tel. no. 

Sharon Babbitt, Director, Knowledge Applications Division (Sharon_Bobbitt@ed.gov) Susan Klein, 
Knowledge Applications Division (Sue_Klein@ed.gov) Stephen O’Brien, Knowledge Applications 
Division (Stephen_0’Brien@ed.gov) 
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Disseminating Promising and Exemplary Programs: 
Planning a System of Expert Panels 

Meeting Minutes 
March 24, 1997 



Opening Welcome and Introductions 

Sharon A. Bobbitt, Director, Knowledge Applications Division 
Office of Educational Research and Improvement 
U.S. Department of Education 

Expert Panel System: An Overview 

J. Stephen O ’Brien 
Acting Team Leader 
Expert Panel System 

Office of Educational Research and Improvement 
U.S. Department of Education 

Legislation: The Educational Research, Development, Dissemination and Improvement Act of 
1994 directed the Assistant Secretary of Educational Research and Improvement, in consultation 
with the National Educational Research Policy and Priorities Board (Board), to establish "panels of 
appropriate qiialified experts and practitioners." These panels are to evaluate educational programs 
and recommend to the Secretary of Education those programs that should be designated as 
exemplary or promising. 

The legislation defines educational programs to include "educational policies, research findings, 
practices, and products." These programs may range in size and complexity fi-om individiial products 
or practices centered on one or more grade levels or target groups to large scale, multigrade 
programs, system wdde initiatives, or statewide curriculum fi"ameworks. 

Standards: The Knowledge Applications Division of the Office of Reform Assistance and 
Dissemination (ORAD) of the Office of Educational Research and Improvement (OERI) has the 
responsibility, in consultation with the Board, to draft regulations governing the System of Expert 
Panels. These draft regulations have been published in the Federal Register for public comment and 
because of substantial changes to the draft will be published again within the month. 

According to the draft regulations. The Department of Education (ED) will establish a standing panel 
of experts and practitioners from across fields to coordinate and monitor the work of the expert 
panels. These standing panel members must possess two or more of the following qiialifications: 

- Demonstrated expertise and experience in one or more specific educational areas; 

- Demonstrated expertise and experience across a broad range of educational policies and 
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practices; 

- Experience in evaluating educational programs; 

- Experience or expertise in product development and dissemination; and 

- Current employment as a teacher, principal or other school-based or community-based professional 
(such as a guidance counselor, school media specialist, health professional). 

ED will also establish a series of expert panels in specific areas. For the past year, two pilot expert 
panels in math/science and gender equity have been working on the issues of process and substance. 
Next year, two new panels will be established, one in technology and one in early reading. Based 
on the recommendations of the standing panel, others will come into being as funds become 
available. In establishing the expert panels, ED will look for individuals who have in-depth 
knowledge of the subject area or content of the program or group of programs to be evaluated. At 
least one current teacher, principal, or other school-based or community-based professional must be 
on each panel and no more than one-third Federal employees. 

Submissions: Any public or private agency, organization or institution, or an individual may submit 
an educational program to the System of Expert Panels for review. The submission will contain a 
description of the program, program materials, and a discussion of the program's (1) evidence of 
effectiveness/success, (2) quality, (3) significance, and (4) usefulness to others. 

Exemplary and Promising Programs: The expert panel will look at how the submission responds 
to the criteria to determine whether the program is designated as exemplary, promising, or neither. 
For a program to be called exemplary, it must be judged excellent in all four of the criteria listed 
above. For example, under the criterion of evidence of success/effectiveness, the results or outcomes 
must be reasonably attributed to the program; and the evidence to support these claims must be 
sustained in multiple sites. 

For ED to designate a program as promising, the program must be identified by the panel as strong 
in all four categories. For example, under the criterion of evidence of success/ effectiveness, the 
program must have defensible overall evidence supporting its claims of worthwhile performance 
results at one or more sites and have evidence of adaptability or transportability to other sites. 

Dissemination: The purpose of the System of Expert Panels is to ensure that the programs 
disseminated by ED are high-quality, research-based programs that have provided evidence 
indicating that they have improved teaching and/or learning or have other worthwhile educational 
outcomes. ED wants to design a dissemination system that makes programs available to the public 
as quickly as possible, that will enable ED to respond to different kinds of requests, and that supports 
the application of research and best practice. 



Mathematics/Science Pilot Panel: Work in Progress 

Patricia O ’Connell Ross 

This pilot panel began work in July 1996. Their goal is to have a process in place to identify 
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promising and exemplary math and scientific programs, products and practices. The panel has 
begun developing criteria and guidelines for the process and will begin field tests this summer. They 
have established two clear thresholds by which to rate materials; (1) fidelity to national standards 
documents for math and science, and (2) evidence of student learning. The panel sees programs that 
are in process as those that may attain the designation of promising and those with established 
backgrounds as exemplary. One main concern the panel has regarding math and science materials 
for consideration is the integrity of the review as the majority of these materials are commercial. In 
regards to dissemination of products, the Panel feels products can be distributed through the National 
Science Foundation and the Eisenhower Program. 



Gender Equity Pilot Panel and American Education Research Association 
Sessions on the Expert Panel System 

Susan S. Klein 

Office of Educational Research and Improvement 
U.S. Department of Education 

The Gender Equity Expert Panel has been facilitated by the Women’s Educational Equity Act Equity 
Resource Center which has complementary responsibilities to help educators learn about and obtain 
high quality gender equity resources. The OERI Assistant Secretary appointed 30 panel members 
with expertise in gender equity from a variety of backgrounds such as researchers, practitioners in 
the schools and communities, developers, technical assistance providers and public officials such 
as members from the Women’s Bureau in the US Department of Labor and a state Department of 
Education. 

Since their initial planning meeting in April 1996, the panel has had eight major accomplishments: 

1 . Developed six subpanels. Supplemental funding has been received for the Gender Equity and 
Disability Subpanel from OSERS and from the Safe and Drug-free Schools Program for the 
Subpanel on the Prevention of Violence and Sexual Harassment. 

2. Expanded the Gender Equity Expert Panel Advisory Group to over 100 experts including 
representatives from other federal agencies, associations, and researchers and teachers with 
gender equity expertise. 

3. Created a listserv to help panel members and advisors to the Expert Panel communicate with 
each other. 

4. Used a variety of ways to call for submissions and invite experts to join the Gender Equity 
Expert Panel Advisory Group. 

5. Developed and revised submission and review criteria and guidelines based on the drafts of the 
OERI core standards for all expert panels. 
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6. Obtained over one dozen submissions on a variety of gender equity programs and started to 
reviev^^ these submissions. 

7. Have been 'working on the development of a review process that will be a fair and positive 
experience for all participants. 

8. Have been trying to follow recommendations from the formative evaluation of the Gender Equity 
Panel on ways to continue progress in serving stakeholders and participants and on making sure 
that the consumer information developed through this effort has a positive impact on helping 
educators make wise choices of tools to promote gender equity. 

The Gender Equity Pilot Panel began in April 1996 with the appointment of 30 panel members from 
a wide range of concentrations. The purpose of the panel is to re-view programs that promote gender 
equity in or through education in order to determine what works in advancing gender equity, to offer 
suggestions that help improve the programs, and to identify and publicize effective programs that 
would be useful to others. 

The panel is di-vided into six subpanels, each dealing with gender equity in a particular area: (1) core 
gender equity, (2) gender equity and disability, (3) mathematics, science and technology, (4) 
prevention of -violence and sexual harassment, (5) teacher education and professional development, 
(6) vocational education and School-to-Work. 

The panel will designate as “exemplary” or “promising” those programs that best accomplish the 
goal of advancing gender equity. The panel’s recommendations, along with summary inf ormation 
on the programs recommended, will be forwarded to the Secretary of Education for approval and 
then -vsdll be publicized via the National Education Dissemination System to help potential users 
obtain information on a variety of promising and exemplary programs. Feedback on programs 
reviewed will also be shared with the submitters so that they will obtain suggestions on how the 
program might be improved. 

Commissioned Papers on Finding and Sharing the Best: A Summary 

Authors of four commissioned papers to be presented in an afternoon session gave brief overviews 
of their paper topics. 

Karen Bogart - Aime Steinmaim Institute 

Examining Federal Approaches Outside the Department of Education to Identify and Disseminate 
the Best 

This paper reviews several federal agencies, including the Federal Emergency Management Agency, 
the Office of Adolescent Pregnancy Programs, the Substance Abuse and Mental Health Services 
Administration, and the Agency for Healthcare Policy Research. The paper makes use of a template 
to examine these different agencies in terms of common dimensions: use of panels, performance 
indicators, rules of evidence, evidence-based centers, use of information technology, and resource 
materials. 
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John Luczak- SRI International 

Learning from Consumer-Oriented Review Efforts in a Wide Variety of Education Organizations 
and Topic Areas 

This paper looks at twelve efforts from five different areas of education including vocational 
education programs or practices, education reviews, educational equity, early childhood education, 
and post-secondary education. The following are the five major conclusions found in the paper: 

1 . There was little differentiation or discussion regarding the differences between “promising” or 
“exemplary” programs or practices in the review efforts studied. Only two of the review efforts 
used the term “exemplary” to describe the programs they selected. 

2. It is most effective to base program evaluation criteria on a careful examination of the research 
based ideas from knowledgeable practitioners or leaders in the field, and current federal 
legislation provisions, whenever possible. 

3. A rating system (only three reviews had one) and/or the use of experts to conduct the reviews 
(five programs used experts in some capacity) made review efforts more credible. 

4. A good application process required a program to complete a self-assessment as part of the 
application process, which led to program benefits before the application was reviewed. Two 
of the applications required or strongly suggested a self-study as a component of their application 
materials. 

5. Three other less frequent processes that appeared to be effective were: (1) the requirement of 
sponsorship by a pre-existing member or promising program; (2) the use of achievement data 
to back up results; and (3) the use of site visits (when costs allowed) to verify information in the 
application. 

Janet Carter- (Former Executive Director of the Bruner Foundation) 

Review of Foundation, Associations, and Non-Profits ’ Practices in Designating Promising and 
Exemplary Programs 

This paper focuses on selected non-governmental efforts to identify what works in and outside of 
education. It asks what might be instructive for OERI’s emerging System of Expert Panels from 
these experiences. 

Key informants were identified from participants in the 1993 Replication Conference. Suggestions 
gained from foundations, non profits and associations were also included. Conclusions reached 
concerning practices of these entities, as distinct from public funders, include; (1) the inclination not 
to reveal the identities of unsuccessful applicants in a review process; (2) staff’s acknowledgment 
of subjectivity in the review process and (3) the impact for many initiatives of having multiple 
funding sources. 

Organizations were found to differ in how they made decisions about promising or exemplary 
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programs. Making a distinction between the two terms was not common. The approaches included: 

• Relying on key staff 

• Using external reviewers 

• Setting up criteria in an RFP and compiling information received accordingly 

• Relying on a separate program development orgaitization who acts for the fimder 

• Employing participatory or qualitative methods to examine particular projects 

• Using data generated from experimental or quasi experimental demonstration projects. 

Four parallel aspects of the work of the Expert Panels with the entities studied were noted: 

1 . An interest in comparative evaluations: As in United Way of America’s commitment to outcome 
measures, the Panels face the need to reassure and counsel those who are to taking part and 
dealing directly with local sensitivities. 

2. The need to gain funding has been met by free standing program developers such as Harvard 
Family Research Project, by demonstrating both research and dissemination capabilities. The 
Panels might want to consider institutional as well as individual memberships. 

3. Commitment to a cause was found to be particularly present in some of the orgaitizations. 
Impact II is both a granting and an advocacy effort for classroom teachers. This lends strength 
to its efforts to find best practices. The Expert Panels ntight consider what would contribute to 
such energy and commitment in their new effort. 

4. Disseminatioras important. The Robert Woods Johnson Foundation builds an audience for its 
announcements and reports by integrating communication professionals with program creators 
and by rewarding potential c\ostomers for reading their RFPs by the significance of the discussion 
of professional questions. 

Summary : All of the orgaitizations in our study rely on staff and expert panels for many of the 
decisions they make regarding prontising programs. In addition those organizations that appeared 
most confident of their conclusions used multiple methods for obtaining information about them. 
Participatory and empowerment evaluations not only helped identify prontising models but also 
generated information that was judged useful to the programs. 

Gary Borich- University of Texas, College of Education 
Advising and Learning from the Combined Review Efforts 

This paper integrates and synthesizes three very varied efforts. The two main focuses of the paper 
were on how to structure the work of authors and broad trends in the integration process. The three 
broad trends consist of establishing the difference between prontising and exemplary designation, 
the use of panels for human insight and judgement, and the integration of quantitative and qualitative 
data. 

Small Group Discussion on Dissemination 

The group was then divided into three smaller discussion groups to address the following three 
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topics: 

1 . What are new ways that the U.S. Department of Education could spotlight exemplary and promising 
practices so that teachers, principals, and superintendents could have informed access to those 
programs and use them effectively? How can practitioners help the Department understand the 
strengths and weaknesses of the programs they are using? Are there different ways to disseminate 
exemplary versus promising programs and if so what are they? What could be done immediately and 
what would need substantial plarming and resources. 

2. What should the U.S. Department of Education do to encourage organizations of every kind to 
participate in its dissemination system? How could these organizations best be involved? What kinds 
of organizations should the Department target as potential best cooperators in this effort? 

3. Given the two-tiered designation of promising and exemplary, what kinds of incentives should the 
U.S. Department of Education encourage and/or provide to entice people to apply for the designation 
of promising for their programs? What kinds of incentives should there be to help people who have 
a promising program move toward the designation of exemplary? What kinds of incentives should 
there be to encourage people to apply for either category? How important are external incentives to 
the system? 

GROUP 1 

Group One focused on question one above. 

Question 1 

• They felt that exemplary and promising practices could be spotlighted on the Internet or within 
consumer reports; both places would advertise contact information for places to find out more 
information about individual pieces. 

• Marketing is needed to make it easier for the public to find out about practices and how to 
implement them. 

• Each exemplary and promising practice could have a short informational video to highlight its 
accomplishments. 

• There could be a consumer report type catalog listing all practices , including short description 
and information on where to find more information about them. 

• Advisory panel members should be used to help disseminate information and market the 
practices. 

• Try to get business and industry to develop or pay for development of Public Service 
Announcements to tell about the Expert Panels’ issues such as gender equity in education, etc. 

• Would like to get professional organizations involved by having them assist in making the call 
for submissions and helping to “advertise” the availability of the consumer reports, such as 
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professional teacher organizations like the National Coxmcil of Teachers of Mathematics. 

• Because such marketing methods could prove costly, beta tests should be run on small samples 
to test effectiveness of a given method of marketing and dissemination. 

• Feedback from each panel should be compiled to compare different methods. Those methods 
that worked best should be replicated throughout the system. All should feel like they are 
partners in the process in order to reinforce each others efforts. Each panel is different and each 
panel has different resources to contribute to dissemination. 

Question 3 

• Possibly focus incentives on the attainment of solid professional development experience 
of money incentives. 

• Promising practices should be given funds to do the work necessary to get to exemplary status; 
this could be a competitive grant program. 

GROUP 2 

Group 2 concentrated on the second of the three questions listed on page six. 

Question 2 

• Because the government does not budget for travel, there are not enough face to face meetings. 
Mailings will not do it. To get good submitters and good people to help with dissemination, 
need regional meetings. 

• Need different kinds or organizations. Include those working in all areas, but not just 
educational organizations, for example, use the American Association of University Women or 
the YWCA. These groups reach many people. Both MADD and AAA have extensive education 
programs. Look at the dissemination systems of other organizations, especially AAA. 

• The guidelines for promising and exemplary programs need examples to capture the attention 
of various agencies. ED needs lots of examples so people can see how it may work for them. 

• The subtleties of promising and exemplary programs must be clearly shown in any 
dissemination. 

• ED needs different kinds of outreach to organizations. 

• If the system goes forward on informal grotmds, it will be very weak. There must be incentives, 
funds, structure. Use technology extensively, but do not think that everyone we want to reach 
has it. 

• Develop lists or directories of community groups we need to communicate with. 
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• Buy into already existing networks. 

• Ease of access to information is an issue. 

• Target state education associations, national education associations like Association for 
Supervision & Curriculum Development; use the Eisenhower system; use the ED national 
education centers. 

Question 3 

• Incentives equal money. Incentives are extremely important and need to be more than oratory 
urgings. 

• Incentives can be prestigious kinds of designations or prizes. 

• Most programs will need funds for replication. 

• If the system of expert panels is to be a true system, it will be symbiotic. The federal 
government has a responsibility to be clearer about evaluation gtiidelines. People need guidance 
from ED on the evaluation of programs. ED needs to be specific about criteria. 

• National Science Foimdation has a handbook that defines what they want. It gives examples of 
the kinds of evidence that NSF expects. ED needs something like that if this system is to work. 

'• Sometimes, different people have different expectations about incentives. Recognition is enough 

for some people; for others it is not. If it will be just recognition, it must be very public and 
prestigious. 

• ED cannot expect individual people to be responsible for disseminating their own projects. ED 
must be the major disseminating force. 

• Reward, funding, and publicity incentives are all essential. 

Question 1 

• There really are not different ways to disseminate promising and exemplary programs. They are 
the same in terms of dissemination. 

• Dissemination should include technical assistance for practitioners. 

• ED needs to tie in to the Comprehensive Technical Assistance Centers. That is an easy way for 
ED to spotlight programs. 

• Spotlighting is not enough. Need to be “proactive” in getting people to use things. 

• There is nothing in it for practitioners to give ED feedback on what the strengths and weaknesses 



are of individual programs. 



GROUP 3 

Group 3 focused their discussion around question three from page six. 

Two main groupings 

1 . Incentives for Exemplary Status 

- Requirement that nominations come from others, instead of an application on one’s own to give 
the nomination a more professional status. 

- The nomination should perhaps come from a minimum of two different fields or agencies with 
which they coimect. 

- Membership of panelists could be by institution instead of by individual to provide for greater 
recognition status. 

- Coimect exemplary status to affirmation and recognition, even accreditation or certification, 
within your field. 

- Provides for a forum to be heard through increased dissemination through the Department of 
Education. 

2. Incentives for Promising Status 

- Applicants would receive assistance in evaluating their work 

- Provides for peer interaction and personal development (also incentive for exemplary status) 

• It is difficult to get people to apply even when incentives are offered, so many question the cost- 
benefit ratio of the time it takes to go through the application process, especially if it involves 
additional research. 

• For many the sense of advocacy (i.e. the desire to spread good practices) serves as an incentive. 

• A major incentive would be the chance for interaction and intellectual exchange with high level 
professionals within your own field. This interaction would also provide for a peer network lets 
you know that others are having the same problems, provides a problem-solving forum. 

• Another incentive for many programs is the opportunity for their practice to be evaluated by the 
panels. 

• One obstacle to getting more people to apply is that many have the drive but not the time nor staff 
to invest in the submission and dissemination process. 

• Need to consider what would happen if people go through the process and do not get the 
“promising” designation. Need to set up a process for the panel to provide insight and assistance 
on areas for improvement so that others will see the value of applying. 

• Some concern was expressed that the criteria are geared to evaluate programs, not products. The 
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panels need to establish processes and criteria for product evaluation that work. May need to 
revise standards for submission of products for some fields. Concern was expressed that some 
fields have no evaluation history and therefore will have little that can meet the criteria. 

• Need to revise standards for submission of products. 

• An incentive for “promising” designation could be linkage to an outsider who can review or 
evaluate their research and data. 

• Some concern about issues of confidentiality: the prestige of panels/reviewers is clearly an 
incentive yet some concern was expressed about sharing their identity. 

• Need to establish guidelines for what is expected of the panel members. A clear, concise process 
needs to be set up regarding submission, evaluation, and acknowledgment. 
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Major Themes Arising From The Three Discussion Groups 



QUESTION ONE: 

Creative Ways to Spotlight Exemplary and Promising Programs 

► Need to Develop a System of Communications that Will Utilize the Benefits of 
Telecommunications and Technology to Spotlight and Promote Selected Model Programs 

• Utilize the Internet by developing resource guides to appropriate sites: 

- Web Pages 

- Chat Rooms 

• Develop and use public service announcements highlighting the research on programs 
and what works and why 

• Develop and distribute clear and concise videos about model programs 

• Develop and distribute resource guides with contacts for each program 

• Develop and distribute a catalogue with a summary of each program and how to 
obtain more guidance and information 

• Develop and distribute consumers’ report guides which allow different aspects of 
programs to be compared 

• Develop a plan to tie programs and their development to activities of Comprehensive 
Technical Assistance Centers 

► Develop guidelines from the expertise of each panel member that will allow constructive 
feedback to compare and contrast programs 

► Utilize systematic methods to test the effectiveness of promising programs 

• Run beta tests on small samples to test the effectiveness of a given method or design 



QUESTION TWO: 

Creative Ways to Involve Various Organizations in the Dissemination Process 

► Need to develop and implement a system for “Face-to-Face” Contacts 

• Hold regularly scheduled regional meetings to allow for peer exchange and learning 

• Develop resource guides of excellent submitters and presenters whose services and 
products can be utilized at these meetings 

12 
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• Develop and utilize resource guides about diverse organizations- YMCA, AAA, 
Organization of University Women, MADD, etc., and study and learn from their 
experiences • 

• Develop and distribute a catalogue that reviews and studies the model programs of 
these diverse organizations that can be utilized to meet key needs for OERI endeavors 

► Need to develop and implement a system to provide information to those without 
technology capabilities 

• Assess the technological needs of the community members and plan 
accordingly 



QUESTION THREE: 

The Use of Incentives to Encourage Applications for Promising and Exemplary 
Programs 

► Acknowledge up-front that resources are limited. However, it is essential to develop a plan 
for the use of incentives to promote the development of programs through the use of various 
rewards, innovative funding, and creative recognition to bestow prestige and opportunities 
for intellectual exchanges. 

• Utilize a flexible process in establishing the ways to nominate programs for 
“promising” and “exemplary” status. 

- Establish guidelines for self nominations 

- Establish guidelines for nominations from peer sources 

- Establish guidelines for nominations from organizations 

• Establish quality guidelines for membership. 

- Develop membership benefits and special funding incentives 

- Hold regular forums to share best practices and research 

- Develop mentorship guidelines and plans for self-development and growth 

► Evaluation criteria and guidelines must be research-based and be clearly and concisely 
stated. Groups need to understand the entire process if “exemplary” and “pro misin g” 
program designations are to have merit and value. 

• Develop a “steps to quality” guidebook utilizing the National Science Foundation 
handbook model. 
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• Develop “troubleshooting assistance” peer network programs and guides to provide 
assistance throughout the submission, evaluation, acknowledgment, and follow 
through process. 

• Provide support for establishing the criteria for product development, evaluation, and 
dissemination. 

• Establish a system to best utilize expert panels and their expertise in all aspects of 
developing quality programs and products and the important issues of confidentiality. 
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Agenda 

System of Expert Panels 
American Educational Research Association 
Regency A, Hyatt Regency Hotel, Chicago 
Monday, March 24, 1997 



9:00 a.m. Coffee 

9:30 Welcome and Introductions 

Sharon A. Bobbitt, Director, Knowledge Applications Division 
Office of Educational Research and Improvement (OERJ) 

U.S. Department of Education 

9:40 Expert Panel System: An Overview 

J. Stephen O’Brien, Acting Team Leader, Expert Panel System, OERI 

9:50 Mathematics/Science Pilot Panel: Work in Progress 

Patricia O’Connell Ross, Team Leader, Javits Program, OERI 

9:55 Gender Equity Pilot Panel and AERA Sessions on the Expert Panel System 

Susan Klein, OERJ Coordinator for the Gender Equity Expert Panel 

10:00 Commissioned Papers and Finding and Sharing the Best: A Summary 

Karen Bogart, Federal Efforts', John Luczak, Education Efforts', Janet Carter, 
Foundation Efforts; Gary Borich, (Synthesis) 

10:10 Introduction to Small Group Discussion on Dissemination (Stephen O’Brien) 

(Each group will try to address all three topics, but first topic in each group will be 
staggered) 

1. What are new ways that the U.S. Department of Education could spotlight 
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could have informed access to those programs and use them effectively? How can 
practitioners help the Department understand the strengths and weaknesses of the 
programs they are using? Are there different ways to disseminate exemplary versus 
promising programs and if so what are they? What could be done immediately and 
what would need substantial planning and resources. 

2. What should the U.S. Department of Education do to encourage organizations of 
every kind to participate in its dissemination system? How could these organizations 
best be involved? What kinds of organizations should the Department target as 
potential best cooperators in this effort? 

3. Given the two-tiered designation of promising and exemplary, what kinds of 
incentives should the U.S. Department of Education encourage and/or provide to 
entice people to apply for the designation of promising for their programs? What 
kinds of incentives should there be to help people who have a promising program 
move toward the designation of exemplary? What kinds of incentives should there 
be to encourage people to apply for either category? How important are external 
incentives to the system? 




11:25 

11:40 



Reports from Discussion Groups 
1 1 :45 Conclusion (Sharon Bobbitt) 
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Summary: System of Expert Panels 
American Educational Research Association 
Regency A, Hyatt Regency Hotel, Chicago 
Monday, March 24, 1997 



Legislation: The Educational Research, Development, Dissemination and Improvement Act of 
1994 directed the Assistant Secretary of Educational Research and Improvement, in consultation 
with the National Educational Research Policy and Priorities Board (Board), to establish "panels 
of appropriate qualified experts and practitioners" to evaluate educational programs and 
recommend to the Secretary of Education those programs that should be designated as exemplary 
or promising. 

The legislation defines educational programs to include "educational policies, research findings, 
practices, and products." These programs may range in size and complexity from individual 
products or practices centered on one or more grade levels or target groups to large scale, multi- 
grade programs, system wide initiatives, or statewide curriculum frameworks. 

Standards: The Knowledge Applications Division of the Office of Reform Assistance and 
Dissemination (ORAD) of the Office of Educational Research and Improvement (OERI) has the 
responsibility, in consultation with the Board, to draft regulations governing the System of 
Expert Panels. These draft regulations have been published in the Federal Register for public 
comment and because of substantial changes to the draft will be published again within the 
month. 

According to the draft regulations, the Department will establish a standing panel of experts and 
practitioners from across fields to coordinate and monitor the work of the expert panels. These 
standing panel members must possess two or more of the following qualifications: 

- Demonstrated expertise and experience in one or more specific educational areas. 

- Demonstrated expertise and experience across a broad range of educational policies and 
practices. 

- Experience in evaluating educational programs. 

- Experience or expertise in product development and/or dissemination. 

- Current employment as a teacher, principal or other school-based or community-based 
professional (such as a guidance counselor, school media specialist, health professional). 

The Department will also establish a series of expert panels in specific areas. For the past year, 
two pilot expert panels in math/science and gender equity have been working on the issues of 
process and substance. Next year, two new panels will be established, one in technology and one 
in early reading. Based on the recommendations of the standing panel, others will come into 
being as funds become available. In establishing the expert panels, the Department will look for 
individuals who have in-depth knowledge of the subject area or content of the program or group 
of programs to be evaluated. At least one current teacher, principal, or other school-based or 
community-based professional must be on each panel and no more than one-third Federal 
employees. 
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Submissions: Any Public or private agency, organization or institution, or an individual may 
submit an educational program to the System of Expert Panels for Review. The submission will 
contain a description of the program, program materials, and a discussion of the program's 
evidence of effectiveness/success, quality, significance, and usefulness to others. 

Exemplary and Promising Programs: The expert panel will look at how the submission 
responds to the criteria to determine whether the program is designated as exemplary, promising, 
or neither. For a program to be called exemplary, it must be judged excellent in all four of the 
criteria. For example, under the criterion of evidence of success/effectiveness, the results or 
outcomes must be reasonably attributed to the program; and the evidence to support these claims 
must be sustained in multiple sites. 

For the Department to designate a program as promising, the program must be identified by the 
panel as strong in all four categories. For example, under the criterion of evidence of 
success/effectiveness, the program must have defensible overall evidence supporting its claims of 
worthwhile performance results at one or more sites and have evidence of adaptability or 
transportability to other sites. 

Dissemination: The purpose of the System of Expert Panels is to ensure that the programs 
disseminated by the Department are high-quality, research-based programs that have provided 
evidence indicating that they have improved teaching and/or learning or have other worthwhile 
educational outcomes. The Department wants to design a dissemination system that makes 
programs available to the public as quickly as possible, will enable the Department to respond to 
different kinds of requests, and supports the application of research and best practice. 
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Papers and Formative Evaluations to Help the Office of Educational Research and 
Improvement (OERI) in the U.S. Department of Education Design: 

• Standards for Designating Promising and Exemplary Programs and 

• A System of Expert Panels to use these Standards 

As of Sept. 1997, papers with dates are available from the Knowledge Applications Division c/o 
Melvin Rogers at melvin_rogers@ed.gov, Tel. 202-219-2149, Fax 202-219-1407. Some of them 
are included in this volume. Others will be included in OERI volumes on Learning from 
Consumer Oriented Review Efforts to Identify and Share the Best and The Gender Equity Expert 
Panel — Its Goals and Early Experiences. 

General Technical Evaluation Assistance: 

Dr. Lois-ellin Datta, Datta Analysis 

68-1769 Lauia Place, PO Box 383768, Waikoloa, Hawaii 96738 
Tel / Fax 808-883-8670, E-mail: 74434.1 122@compuserve.com 

Dr. Michael Scriven 

PO Box 69, Point Reyes, CA 94956 

Tel. 415-663-1511, Fax 415-663-1913, E-mail: scriven@aol.com 

Drs. Datta and Scriven have completed their work which consisted of much timely advice as OERI staff 
developed standards for promising and exemplary programs. Some of their advice is available in a recent 
paper “Incorporating Research Findings and Practice in Expert Panel Work: A Dialog Between Michael 
Scriven and Lois-ellin Datta ” 1/31/97 which was distributed Feb. 12, 1997 on the findbest and gndrpan 
listservs and to the mathematics and science expert panel members. 

Specialized Technical Evaluation Assistance: 

Towards Standards for Identifying Promising Design-Based Whole School Reforms 

Dr. Tom Glennan and Dr. Sue Bodilly 

Rand, 2100 M. Street, N.W., Washington, DC 20037-1270 

Tel. 202-296-5000 ext. 5377 for Bodilly, Fax 296-7960, E-mail: bodilly@wash.rand.org 
“Towards Standards for Identifying Promising Design-Based Whole School Reforms”. Oct. 1996. 

Evaluation and Standards for Schoolwide Programs and Other Systemic Initiatives 
Dr. John Hollifield, Johns Hopkins University 
Home: 1738 Grange Road, Baltimore, MD 21222 

Tel: 410-516-8810(0), 410-284-7794 (H), Fax 410-516-8890, E-mail: jhollifi@inet.ed.gov 
“Evaluation and Standards for Schoolwide Programs and Programs Conducted in Multiple 
Sites,” John H. Hollifield, Samuel C. Stringfield, and Rebecca Herman, Jan. 1996 

Methods of Evaluating Complex Mathematics and Science Programs 

Dr. Walter G. Secada, School of Education, University of Wisconsin-Madison 

1025 W. Johnson Street, Madison, WI 53706 

Tel. (608) 263-4544 Fax (608) 263-4506 

E-mail: secada@vms.macc.wisc.edu 

Reviews and Analyses of Systematic Efforts to Identify and Share What Works: 

Building on Review Efforts in Mathematics and Science Education 
Carol Muscara, Computer Technology Services, Inc. 

1700 Rockville Pike #315, Rockville, MD 20852 

Tel. 301-468-1 160, Fax 301-231-71 19, E-mail: cmuscara@eats.com 

“A Discussion of Some U.S. Efforts for Programs and Resources in Mathematics and Science”, Nov. 

1996. 



Commisioned Papers to Help Design a System to Designate Promising & Exemplary Education Programs 

Learning from Consumer-Oriented Review Efforts in a Wide Variety of Education Organizations and 
Topic Areas, July 1997 

John Luczak has become the primary author with advice from Joan Ruskus, SRI International 
16 1 1 N. Kent Street, Arlington, VA 22209 

John is now working in the Office of the Deputy Secretary, U.S. Dept, of Education 
Tel. 202-205-7848, E-mail: john_luczak@ed.gov 

Review of Foundation, Associations, and Non-profits Practices in Designating Promising and 
Exemplary Programs by Janet Carter and Diane Schilder, July 1997 
Dr. Janet Carter (Former Exec. Director of the Bruner Foundation) 

E-mail: jcarterl2@aol.com 

Examining Federal Approaches Outside the Department of Education to Identify and Disseminate the 
Best, June 1997 

Dr. Karen Bogart, Anne Steinmann Institute 

1027 Riva Ridge Drive, Great Falls, VA 22066 

Tel 703-759-5472, Fax 703683-4915, E-mail: kbogart@erols.com 

Standards and their Use in the Food and Drug Administration (FDA) 

Dr. Gerald (Jake) Barkdoll, DPA 
33 Vista De Oro Road, Placitas, NM 87043 
Tel: 505-867-2899, Fax: 505 867-2922 

“Standards and Their Use in the Food and Drug Administration” Jan. 19, 1996 

Advising and Learning from the Combined Review Efforts 
Dr. Gary Borich, Univ. Of Texas, College of Education 
Austin, Texas 78712 

Tel. 512-471-4155 W, 441-2082 home. Fax 512-471-1288, E-mail: garyborich@mail.utexas.edu 
“A Synthesis and Integration of U.S. Evaluation Efforts to Identify Promising and Exemplary 
Educational Programs, Products and Practices” July 1997 

Formative Evaluations of Pilot Expert Panels: 

Formative Evaluation of the Pilot Expert Panel in Gender Equity 

Dr. Patricia B. Campbell, Campbell-Kibler Associates 

80 Lakeside Dr., Groton, MA 01450 

Tel. Fax 508-448-5402, E-mail: ckassoc@tiac.net 

“First Year Formative Evaluation: The Pilot Expert Panel in Gender Equity,” Nov. 1996. 

Formative Evaluation of the Pilot Expert Panel in Mathematics and Science Education 

Dr. Senta Raizen, Director, National Center Patti Bourexis, The Study Group, Inc. 

for Improving Science Education 209 Sir Walter Raleigh Dr, 

2000 L Street, N.W., Suite 603, Washington, DC 20036 Kill Devil Hills NC 27928 

Tel. 202-467-0652, Fax 202-467-0659 Tel. 919-441-2788, Fax 919-441-9663 

E-mail: raizen@ncise.org or callan@ncise.org E-mail: studygroup@aol.com 

Pilot Expert Panel Contacts: 

Pilot Expert Panel in Gender Equity, Katherine Hanson, Director 
Women’s Educational Equity Act Publishing Center, Educational Development Center 
55 Chapel St. #200, Newton, MA 02160, 800-225-3088, or 617-969-7101, Fax 617-332-4318 
E-mail: Katherine Hanson, katherinh@edc.org or Debra Whitcomb, debraw@edc.org 
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Commisioned Papers to Help Design a System to Designate Promising & Exemplary Education Programs 

Pilot Expert Panel in Mathematics and Science Education, Pelavin Research Institute, 1000 Thomas 
Jefferson Street, NW, Suite 400, Washington, DC 20007 

Tel. 202-944-5396 Lynn Fender, Amy O’Malley 944-5366, Fax 202-944-5454, E-mail: ao’malley@air- 
dc.org, lfender@air-dc.org 

Other 1997 Papers: 

“Department of Education Draft Standards for Conduct and Evaluation of Activities Carried Out by the 
Office of Educational Research and Improvement (OERI)— Designation of Exemplary and Promising 
Programs” (Public version of draft shared with the OERI National Educational Research Policy and 
Priorities Board, March 21, 1997.) The final draft should be published in the Federal Register for public 
comment soon. (Contact: Dr. Stephen O’Brien, U.S. Department of Education, 555 New Jersey Ave, 
NW. Room 502B, Washington, DC 20208-5643, Tel. 202-219-2141, E-mail steve_o’brien@ed.gov) 

“Expert Panels for Promising and Exemplary Innovations: A “Fine Idea From the Feds” by Peter 
Stidmzn, NCRVE Centerwork: National Center for Research in Vocational Education, Univ. of 
California, Berkeley", Vol. I No. 1, Winter 1997, pages 5-7. (Contact: Dr. Peter Seidman, Director of 
NCRVE Dissemination Program, NCRVE, 2030 Addison St., Suite 500, Berkeley, CA 94720-1674, Tel. 
800-762-4093, E-mail: seidman@uclink.berkeley.edu) 

“A System of Expert Panels and Design Competitions: Complementary Federal Approaches to Find, 
Develop and Share Promising and Exemplary Products and Programs” by Susan S. Klein. Educational 
Researcher, AugJSept. 1997, p. 12-20. 

“An Overview of OERI Efforts to Develop A System to Designate and Disseminate Promising and 
Exemplary Products, Programs and Practices” by Susan S. Klein. This is similar to the above paper, but 
shorter. March. 19, 1997. Presented at AERA Annual Meeting, Chicago. 

“Design Competitions and Expert Panel: Similar Objectives, Very Different Paths” by Robert Slavin, 
Educational Researcher, Aug./Sept. 1997, p. 21-2. 

“Presenters Comments: Symposium on the Gender Equity Expert Panel”, AERA Annual Meeting, 
Chicago, March 25, 1997. 

“The Gender Equity Expert Panel Initiated by the U.S. Department of Education,” by Susan Klein, 
Presented at the National Coalition for Women and Girls in Education Annual Conference, Maui, 

Hawaii, July 21, 1997. 

“How the Gender Equity Expert Panel Can be a Gem” by Lois-ellin Datta, Presented at the National 
Coalition for Women and Girls in Education Annual Conference, Maui, Hawaii, July 21, 1997. 

“Disseminating Promising and Exemplary Programs: Planning a System of Expert Panels, Meeting 
Report and Follow-up Papers”. Includes papers by John Evans, Christine Dwyer, John Luczak, Gary 
Borich, Janet Carter, Floraline Stevens, Karen Bogart, Lynn Fox, Harilyn Rousso, Marylin Hulme, Mary 
Wiberg, Patricia Ortman and Sue Klein (Should be available by Oct. 1997). 

Earlier Related Commissioned Papers: 

“Whose Knowledge is It?: Involving Teachers in the Generating and Using of Information on 
Educational Innovations”, Patricia B. Campbell, Campbell-Kibler Associates, July, 1994 

“A Matter of Consensus”, Lois-ellin Datta, Datta Analysis, Aug. 1994 
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Commisioned Papers to Help Design a System to Designate Promising & Exemplary Education Programs 

“Readiness for Change, Educational Innovations, and Educational Reform”, Thomas E. Backer, Human 
Interaction Research Institute, Feb. 1995. 

“Technical Assistance and the Creation of Educational Knowledge” by Brenda J. Turnbull, Policy 
Studies Associates, March, 1996. 

A series of shorter commissioned papers from work at the March 1997 planning session on the System of 
Expert Panels should be available by Aug. 1997. 

Primary OERI Contacts: 

For Mathematics and Science Education Work of Muscara, Secada, Raizen and Bourexis: 

Dr. Patricia Ross, OERI Liaison to the Mathematics and Science Expert Panel 
555 New Jersey Ave., N.W., Room, Washington, DC 20208-5645 
Tel. 202-219-2064, Fax 202-219-2135 
E-mail patricia_ross@ed.gov 

For all the Gender Equity Expert Panel. 

Dr. Susan S. Klein, OERI Liaison to the Gender Equity Expert Panel 
555 New Jersey Ave., N.W., Room 508 F, Washington, DC 20208-5643 
Tel. 202-219-2038, Fax 202-219-1407 
E-mail sue_klein@ed.gov 

For the System of Expert Panels 

Drs. Sharon Bobbitt and Stephen O’Brien 

555 New Jersey Ave., N.W., Room 508, Washington, DC 20208-5643 
Tel. 202-219-2126, 219-2141, Fax 202-219-1407 
E-mail sharon_bobbitt@ed.gov, steve_o’brien@ed.gov 
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A Closer Look at Consumer-Oriented Review Efforts: 
Methodology and Lessons Learned from a Review of a 
Wide Variety of Education Organizations and Topic Areas 



John Luczak and Joan Ruskus, SRI International 
Prepared for U.S. Department of Education 
Office of Educational Research and Improvement 
Paper presented at AERA Session 5.26, March 24, 1997 
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Background and Organization 

OERI has begun conducting research on existing review efforts that attempt to identify 
and share the best of what is available with consumers. This paper was commissioned to 
review efforts in a wide variety of education organizations and topic areas, excluding 
review efforts in the areas of mathematics and science programs. The 12 review efforts 
focused on a wide variety of topics in education: five reviews evaluated vocational 
education programs or practices, three were special education reviews, two concentrated 
on educational equity, one targeted early childhood education, and one other review effort 
focused on postsecondary education. 

Identifying Review Efforts 

The process to identify review efforts was not systematic in a research sense. OERI made 
initial suggestions of certain review efforts and provided names of important contacts who 
were able to identify other promising efforts (similar to snowball sampling). The principal 
criteria for selection was representing a wide range of review efforts with as little 
duplication as possible. 

Types of Resources Reviewed 

Most of the efforts (two-thirds) examined reviewed programs rather than products or 
practices. The programs reviewed were diverse and included the following: advanced 
technology centers, transition programs, high schools combining challenging academic 
courses and modem vocational studies, career guidance and counseling programs, state- 
level special education programs, girls’ achievement and healthy development 
strategies/programs, early childhood programs, and higher education reform programs. 

Of the remaining efforts, one analyzed School-to-Work instmctional products while 
another effort evaluated youth development/employment practices. Another review effort 
created a generic screening process for use with any type of product, practice, or program. 
And finally, one review process involved five or six separate review efforts and therefore 
examined all three types of resources. 



Types of Review Efforts 

There was significant diversity among the 12 review efforts in terms of approach used to 
review materials. The approach was often determined or influenced by 1) the review 
process itself, and 2) the number of programs or practices that needed to be reviewed. 



The following is a mini-typology (with some overlap) of the different review effort 
approaches: 



Formal rating system used with point totals in review process (3 review efforts) 
Program applications reviewed by experts (2), or reviewed by experts after a first 
round of review by project staff (3) 
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Evidence of Success 

The evidence of success required to become a proraismg program or practice varied 
considerably by type of review effort. A few of the review efforts (the transition and 
higher education reform programs in particular) placed an extreme emphasis on high 
quality evaluation data, preferably linked to targeted outcomes. While not eveiy program 
had a plear project description, evaluation design, and a^summaiy report of evaluation 
findings, most of the appUcations tried to ensure that some sort of baseline evaluation 
standard was met. 

One review effort (the high schools that combine challenging academic courses and 
modem vocational studies) used its own achievement data — based on NAEP — to monitor 
student progress in membership schools. Another application required a separate narrative 
explaining the evidence of the program’s effectiveness. But most fi"equently, review 
efforts required applicants to indicate how they met the standards and indicators used as 
effectiveness criteria. 

Incentives for Programs 

Some of the review efforts were better than others at providing incentives for programs or 
practices to apply for promising or exemplary status. Programs were apt to pose the 
following questions when considering application; What do we get after we are reviewed? 
Do we get to share resources with other selected programs? How often do the selected 
programs talk to each other? 

Many of the review efforts disseminated information to programs in the following ways: 

• At least five efforts had conferences, and one effort (the advanced technology centers) 
required participation in a conference at least once a year. 

• For many review efforts, newsletters were the most popular vehicle for publicity and 
sharing between programs. 

• At least five of the review efforts set up program profiles, in either a report, book, or 
database format. 

• At least two efforts established home pages on the Web and used the site to share 
information, answer questions, etc. 

• One review effort sent out a press release of its “exemplary” programs to over 200 
professional newsletters and the selected institution for publicity purposes. 

Membership programs (there are three such review efforts) were more likely to receive 
some of these dissemination benefits because they fi-equently pay a membersWp fee 
(between S5 00-$ 1,000 for two of the review efforts). They were also more likely to 
participate in self-assessment/professional development activities. Two review efforts (the 
ea rly childhood programs and youth development/employment practices) required or^ 
strongly suggested the completion of a self-assessment before completing an application. 
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• Applications reviewed by team of staff with semi-structured process in place (4) 

• Programs or practices reviewed informally by small team or single staff member (1) 

• Generic screening process developed for a diverse group of users (1) 

Almost all of the review efforts had a set number of program applications to evaluate. 
Many reviewed materials sent to them through a call for applications or nominations, 
while others received a consistent number of membership applications each year. Only 
two efforts conducted their own searches for promising practices and programs. The 
search for girls’ achievement and healthy development strategies/programs involved a 
review of more than 500 studies and reports. The other effort involved five or six 
separate reviews conducted by staff who each put together (in a particular subject area) a 
list of at least ten promising programs fi'om the multitude that existed. 

Evaluation/Effectiveness Criteria 

Almost all of the review efforts had evaluation or effectiveness criteria and used them to 
evaluate prospective programs or practices. The evaluation/effectiveness criteria for 
programs obviously differed depending on the subject area and goals of the specific review 
effort. Still, some similarities emerge looking across the 12 review efforts. 

The most common non-subject area specific review criteria were; 1) program 
organization and management; 2) collaboration (with other agencies, family, etc.); 3) 
student, youth, or client focus/development (life skills oriented, especially with vocational 
and special education programs); 4) equity and diversity needs; and 5) program evaluation 
and evidence of success. These standards were often associated with indicators (anywhere 
firom 2-11 per standard) that provided detailed benchmarks on how to meet the standards. 

It is also notable that some efforts emphasized replicability as a criteria more than others. 
The generic screening process and the two review efforts involving secondary and 
postsecondary institutions emphasized understanding their context as well as any 
program’s context that was being considered as a model for implementation. 

An important finding is that these criteria appeared to be most effective when they were 
based on: 

• Research findings or a literature review of the current research (3 review efforts) 

• Extensive feedback fi'om practitioners/professionals/leaders in the field (3) 

• Provisions of current federal legislation (2) 

Conducting site visits to “promising” programs before making a final decision about their 
status was a luxury that only three of the wealthier review efforts could afford. An expert 
review team — which five review efforts took advantage of in some format — often 
conducted or helped with on-site visits. These e,xpert review teams were always made up 
of 3-5 people. 
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Lessons Learned for “Promising” and “Exemplary*’ Standards and Expert Panels 

• There was little differentiation or discussion regarding the differences between 
“promising” or “exemplary” programs or practices in the review efforts studies. Only 
two of the review efforts used the term “exemplary” to describe the programs they 

’ selected. 

• It is most effective to base program evaluation criteria on a careful examination of the 
research base, ideas from knowledgeable practitioners or leaders in the field, and 
current federal legislation provisions, whenever possible. 

• A rating system (only three reviews had one) and/or the use of experts to conduct the 
reviews (five programs used in some capacity) made review efforts more credible. 

• A good application process required a program to complete a self-assessment as part 
of the application process, which led to program benefits before the application was 
reviewed. Two of the applications required or strongly suggested a self-study as a 
component of their application materials. 

• Three other less popular processes that appeared to be effective were: 1) the 
requirement of sponsorship by a pre-existing member or promising program; 2) the use 
of achievement data to back up results; and 3) the use of site visits (when costs 
allowed) to verify information in the application. 
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Review of Foundation, Associations and Non-Profits Practices 
In Designating Promising and Exemplary Programs 
by Janet Carter and Diane Schilder 



The paper focuses on selected non-govemmental efforts to identify what works in and outside of 
education. It asks what might be instructive for OERR’s emerging System of Expert Panels from 

these experiences. 

Key informants were identified from participants in the 1993 Replication Conference and 
su«raestions gained for foundations, non-profits and associations to be included. Ten are 

outlined. 

Conclusions reached concerning practices of these entities, as distinct froin public funders, 
include: (1) the inclination not to reveal the identities of unsuccessful applicants^ in a review 
process; (2) staff’s acknowledgment of subjectivity in the review process; and (3) the impact for 
many initiatives of having multiple funding sources. 

Diversity of methods were reported in the organizations’ attempts to determine ‘‘best practices.” 
Criteria forjudging the quality of different efforts were based on the internal vision and mission 
of the foundation or non-profit. Best practices were identified by expert panels by setting up 
funding systems with publicly stated standards and by awarding recognition to innovative 

practices. 

Organizations were found to differ in how they made decisions about promising or exemplary 
programs. Making a distinction between the two terms was not common. 

The approaches include: 

1. Relying on key staff 

2. Using external reviewers 

. 3. Setting up criteria in REP and compiling information received accordingly 
. 4. Relying on a separate program developing organization who acts for the ^der 

5. Emplo^g participatory or qualitative methods to examine particular projects 

6. Using da ta generated from experimental or quasi-experimental demonstration projects 

Four parallel aspects of the work of the Expert Panels with the entities ^^ed were noted: 

1. An interest in comparative evaluations: As in United Way of America’s commitment to 
outcome measures, the Panels face the need to reassure and counsel those who are to take 
part and deal with local sensitivities. 

2. The need to gain funding has been met by free standing program developers such ^ Harvard 
Family Research Project by demonstrating both research and dissemination capabilities. The 
Panels might want to consider institutional as well as individual memberships. 

Commitment to a cause was found to be particularly present in some of the organizations. 
Impact n is both a granting and an advocacy effort for classroom teachers. This lends 
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strength to its efforts to i5nd best practices. The Expert Panels might consider what would 
contribute to such euCTgy and commitment m their new effort 

4. Dissemination is important and the Robert Woods Johnson Foundation builds an audience for 
its annomio pmmts and reports by integrating communication professionals with program 
creators and by rewardin g potential customers for reading their RFPs by the significance of 
the discussion of professional questions. 

Summary: All of the organizations in our study rely on staff and expert panels for many of the 
decisions they maVo regarding promising programs. In addition those organizations that 
appeared most confident of their conclusions used multiple methods for obtaining information 
about them Participatory and empowerment evalxmtions not only helped identify promising 
models but also generated information that was judged useful to the programs. 
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Bender Equity in Biucation 

Have you developed programs and projects that support gender equity in education? 

Have you ever wished you could share your innovative ideas with other educators 
throughout the nation? Have you ever wanted the opportunity to ask others how they 
teach a particular course, develop and implement policy, or provide professional develop- 
ment in ways that support the learning of girls and boys? Have you looked for materials 
that will help you make sure your classes or programs are gender equitable and been 
unable to determine whether the materials you find acmally worked? 

If you answered “yes” to any of these questions, you can support and receive help from 
the National Expert Panel on Gender Equity. You may also receive national recognition 
for your work. 

Wliat Is the Hatioiial Gender Equity Expert Panel? 

OERI in the US Department of Education is pilot testing an expert panel on gender equity 
to help the Secretary of Education designate promising and exemplary products, pro- 
grams, and practices that focus on promoting gender equity. The panel is facilitated by 
the WEEA Equity Resource Center, located at Education Development Center Inc. 

(EDC). In this pilot test the Expert Panel will form six subpanels: 

• core gender equity 

• school-to-work/vocational education 

• teacher professional development 

• combating violence/sexual harassment 

• disabilities 

• mathematics/science/technology 




To continually improve education in the United States, we need to find ways that work to 
promote the academic achievement of all students — paying particular attention to gender 
equity. The main purpose of the Gender Equity Expert Panel is to provide educators with 
a central source through which they can locate policy, practice, and instructional materi- 
als they need for gender fair education. The expert panel will collect, review, and recom- 
mend promising and exemplary products. EDC, in collaboration with the Department of 
Education, will disseminate information about these materials that show “what works” to 
teachers and administrators nationally. 



BEST COPY AVAILABL 
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How Can Yon Contribiite to the Gender Equity Expert Panel? 



We welcome your recommendations and submissions. The expert panel will focus 
primarily on programs and products that have immediate application to increasing gender 
equity in schools, colleges and universities, and community organizations. These will 
include: curriculum, program models, implementation models, textbooks, and similar 
materials that can be used directly by educators and administrators. Other teachers will 
have the opportunity to benefit fiom your original materials. You will also have the 
opportunity to access a collection of resources through the WEEA Equity Resource 
Center. » 

What Are the Procedures for Submitting Materials? 



Submissions to the Gender Equity Expert Panel need to be submitted with the guidelines 
and program information. These guidelines can be obtained from the WEEA Equity 
Resource Center world wide web site or by contacting the center. Send your submission 
with four copies of your materials to the Center. They will forwarded to the appropriate 
subpanel chair for the review process. As part of the review, the subpanel may contact 
you for further information or clarification and you will be informed of the final recom- 
mendations. Those materials recommended as exemplary or promising will be listed on 
the web site and in a series of documents. If your products are selected, you will also 
receive a formal acknowledgment that can be used to promote your work. 



Types of Materials 

Materials can be print or non-print items, including multimedia formats such as software, 
videos, or audiotapes. (If you are submitting non-print items, you can send fewer copies.) 



Impact Information 

We would like you to include examples to show the impact/success of your program. 

This could be a formal evaluation or longitudinal study. Or it could be other indicators 
such as participant evaluations, reviews in local media, or indications of shifts in thinking 
or behaviors. 



Original Materials 

We are looking for original materials. If you have adapted earlier materials or programs 
developed by others, please indicate this in your descriptions, including descriptions of 
how you changed the approach or materials. 
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We look forward to your submissions! 

For further information or to submit your programs contact either of the following: 




Joe Maxwell, Sr. Researcher e-mail: jmaxwell@edc.org 

WEEA Equity Resource Center 
Education Development Center, Inc. 

S5 Chapel St. 

Newton, MA 02158 

phone: 617-969-7100 Fax:617-332-4318 

Web site: http7/www.edc.org/CEEC/WEEA/pubs/expanel/paneldescr.html 



Susan 



Klein, OERI, 



202-219-: 



20^1 



fax 202-219-1407 



or sue_klein@ed.gov 



GUIDELINES FOR SUEMTITING PROGRAMS TO THE GENDER EQUITY 

EXPERT PANEL FOR REVIEW 



Goals Of The Gender Equity Expert Panel 

The "Gender Equity Expert Panel" is one part of a proposed system of expert p^els bemg 
developed under the "Educational Research, Development, and Dissemmauon A« of 1^9^, the 
authoring legislation of the Office of Educational Research and Improvement (OERI) ot ^ 

Department of Education. (See References) The purpose of the panel is to re^ew prograim that 
promote gender equity in or through education in order to determine what works m advancing 
gender equity, to offer suggestions that help improve the progr^, and to identify and 
effectiveVograms that should be useful to others. (As defined m the OERI legislation, program 
includes products, practices, policies, and research findings.) 

The panel will designate as "exemplary" or "promising" those programs that best accomplish Ae 
goal of advancing gender equity. (The proposed OERI regulations govemmg &e expen panels 
roecify that the major distinction between promising and exemplaiy progr^ is that exempt^ 
programs should have evidence that they work in multiple contexts and with muluple populauons, 
while promising programs may have been used in only a single site.) The panel s 
recommendations, along with summary information on the programs reco^ended, wih oe 
forwarded to the Secretary of Education for approval and then will be pubhcized via the Nauonal 
Education Dissemination System to help potential users obtain irtform^on on a variety of 
promising and exemplary programs. Feedback on programs reviewed _wm dso be shared with the 
submitters, so that they will obtain suggestions on how the program might be improved. 



The Review Process 

The Gender Equity Expen Panel is divided into six subpanels, each dealing with gender equity in a 
particular area: 



Core Gender Equity 
Gender Equity and Disability 
Mathematics, Science, and Technology 
Prevention of Violence and Sexual Harassment 
Teacher Education and Professional Development 
Vocational Education and School-to-Work 



The information provided by submitters in response to these guidehnes, and ±e acmal program 
materials and supporting documents such as evaluation reports and case smdies, wm be revwwed 
bv at least two members of the appropriate subpanel or advisors selected by the subpanel. Uie 
reviewers may also request clarification or additional information fi-om the submitters, (bucn 
requests will be coordinated by the subpanel chair.) 

This review is confidential; reviewers will not be publicly identified. Reviewera' comments will 
then be discussed by the expert panel members, and a decision reached aboutwhether or not the 
program will be recommended for designation as promising or exemplary. The resists of the 
reviewers' discussions will be synthesized and shared with submitters, and may be mcorporated in 
a public program description if the program is recommended as exemplary or promismg. 
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What Programs Are Appropriate For Review by the Gender Equity Expert Panel? 

In order for a program to be appropriate for submission, it must meet all three of the following 
requirements: 

1 It addresses the goal of advancing gender equity in or through education. (This does not 
nwd to be the primary goal of the program, but it must be a significant purpose or outcome.) 

NOTE: OERI has also established a second expert panel on mathematics and science education, 
and is developing a procedure to handle submissions that are not appropriate for these two expert 
panels. Contact Sharon Bobbitt at OERI (Sharon_Bobbitt@ed.gov) for information on these 

options. 

2. It could be adopted or adapted and used by others. 

3 There is some evidence that the program has had a positive irnpact on gender equity for 
participants, students, or users in at least one setting or location. Tlus 

necessLily need to come from a formal evaluation of the program; it could ^ 

from participants, developers' experiences with program outcomes, or other kmds of evidence, 

either quantitative or qualitative. 



How To Submit A Program For Review 

Please provide the information requested in the Cover Sheet at the end of thpe Guide^es, ^d m 
Parts I md n of the Guidelines, as well as copies of program inatenals, eyaluauon reports, case 
studies etc. We anticipate that submission statements (excluding supporting matenals) wul not 
need to be longer than 10-15 single-spaced pages. If possible, please send four copies of *e 
submission statement and supporting materials, and include melectromc version of *e submission 
statement- the latter can be sent either on disk or by e-mail. (Other iMtenals can also be sent 
electronically, if this would be more convenient for you.) If you prefer, you can use &e clectromc 
version of the Guidelines (available on the Web site listed below, or we can e-mail it to you), and 
simply insert your responses after each question or criterion. 

Completed submissions should be mailed to: 

Gender Equity Expert Panel, atm. Joseph A. M^well, Semor Research ^sociate, "^EA Equity 
Resource Center, Education Development Center, 55 Chapel St., Newton MA 02160. 

If YOU have questions about the Guidelines', the submission procedmes, or the review process, 
please contact Joe Maxwell at 617-969-710)0 x2467, fax 617-332-4318, e-mail 
<jmaxwell@edc.org>; Sue Klein at ^2-219-2038, fax 202-219-1^ ,e-m_ p . 
<iue.klein@ed.gov>; or the WEEA ^uity Resource Center at 800-225-3088 
information on the expert panel guidelmes and review process is available on the EDC Web site. 

http://www.edc.org/CEECAVE^weeainfo/paneldescr.html 

Part I: Program Background Information. 

1 . Program Abstract (250 words or less). This should briefly summarize the program’s 
goals, intended audiences, and how the goals are accomplished. 

2. Description of the Program: Describe the program's purposes, philosophy and history, 
funding and staffing, and important, innovative, or unique features. 
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3 . Site Information and Demographics: Describe briefly where and how the program has been 
used and by what types of users. If possible, include ethnic, racial, and gender percentages. 

4. Contact information (if program is recommended, this information will enable a potential 
user to obtain or access the program): Organization, contact person, address, telephone and fax 
numbers, e-mail address and WWW home page if available. 

Part II: Responses To The Review Criteria ' 

Below are the criteria that the panel will use in deciding if a program shodd be recommend^ ^ 
exemplary or promising. In addressing these criteria, please respond to individual cntena (a, b, c, 

d e etc ) rather than simply to each of the numbered standards (1-4). Responses to the cntena 

will’ be seen as particularly credible if they acknowledge weaknesses as well as s^n^. and 
specify how they plan to address the weaknesses. (For example, under Part n. Section 2^ ii 
statistics on workforce participation by women have changed since &e current vereion of the 
program submitters can explain that they plan to update their statistics before their next pnntmg, or 
that they’ will indicate where future users could quickly Md easily fed current informauon.) wtme 
the reviewers will be basing their assessment of these criteria to a sigi^cant extent on the n^en^ 
submitted, it would be helpful if you briefly respond to each of the criteria, emphasizing poi^ that 
might be overlooked and noting relevant pages in supporting materials where appropriate. This is 
your opportunity to address concerns that reviewers may have in examining the materials 
submitted, as well as to highlight particular strengths of the program. 

1. Evidence of Success/Effectiveness in Promoting Gender Equity 

1 a. What evidence do you have that the submitted program has been successful or effective in 

addressing problems of gender equity in at least one site? Evidence may include evaluahon results, 
information from users, evidence based on a range of assessments, or other mdicators that you 
believe are appropriate. You may refer to evaluation reports or other documents that you send to 
support your responses. If you believe that the program is beneficial for multiple racial, ethmc, 
etc. groups, and for both males and females, please provide data on program outcomes that have 
been disaggregated by group and sex. 

If you feel that the submitted program could be recommended as exemplary in advancing gender 
equity, please also respond to lb and Ic. 

lb. What evidence do you have that the submitted progr^ has been successful or ^ective in 
adi-essing problems of gender equity for multiple sites or with multiple populations? Please 
discuss any evidence which suggests that the program may not be beneficial for specific 
populations. 

1 c. What evidence do you have that the program is as good or better than other gender equity 
programs with similar purposes? 

2 . Quality of the program. 

2a. In what ways is the program based on sound research and practice? 

2b To what extent are both equity information and subject content accurate and up-to-date? 

2c. What advantages does the program have over existing alternatives? How does it 
complement other programs? 



2d. In what ways does the program promote equity with respect to, and how is it free of bias 
based on, race, gender, age, culture, ethnic origin, disability, or limited English proficiency status? 
(This may include realistic and non-stereotyping portrayals, inclusive respectful language, and 
showing a wide range of roles and attributes.) How does it foster high expectations for the 
success of all participants? How does it relate to compliance widi federal civil tights laws such as 
TiUeIX? 

2e. In what ways is the program appropriate, engaging, and motivating for the intended 
audiences? 

2f. Do the TTiaferiak associated with the program conform to accepted s^^ds of technical 
quality? For example, do they adhere to national computer software or audiovisual standards? Are 
they physically durable? Are they easily usable by the intended users (including users with 
disabilities)? 

3 . Educational Signiflcance. 

3 a. How does this program contribute to solving or alleviating significant educational problems 
of gender equity? 

3b. In what ways does it address federal educational responsibilities in this area? 

3c. Does the program contribute to increased knowledge of educational problems or issues, or 
improved strategies for teaching or learning? 

4 . Usefulness to Others 

4a. Is the program reasonable in terms of costs to potential users, in relation to the expect^ 
benefits and compared to other alternatives? (Costs include money, staff time, and other required 
resources.) 

4b. Is the program easily available to potential users, and is it described clearly enough so that 
it can be adapted or adopted in new sites? Are there legal or practical constraints on its use such as 
copyright restrictions or a particularly large scope? Is it likely that others can implement the 
program to obtain the same positive results as the original developer? 



References 

The criteria and procedures described in these guidelines are based on the draft of the proposed 
OERI regulations governing the expert panels and the proposed standards for designating 
promising and exemplary programs (Department of Education Regulation 34 CFR Part 701, 
Federal Register, June 3, 1996). See also Sue Klein, "Highlights of the Proposed Consumer- 
oriented Evaluation Component of the National Education Dissemination System, NEDS" 
(unpublished manuscript, April 1996) and Katherine Hanson, "Gender %uity Expert Panel" 
(unpublished concept paper), both available from Sue Klein at OERI. 
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Submission Cover Sheet, Gender Equity Expert Panel 



Program Title: 

Name of submitter and organization: 




Address: 
Telephone: 
TTY/TDD: 



Fax: 
E-mail: 



WWW home page: . 

t 

Date of Submission: 



List of 3ad attachments included with submission: 



Submitter’s Involvement with Program: 
Author/Developer Publisher 



Evaluator 



User 



If there are different versions of the program currendy available, explain how what you are 
submitting differs from others. 



Subpanel(s) that should review the program: 



Core Gender Equity 

Mathematics, Science, and Technology 

School-to-Work/Vocational Education 
~ Teacher Education and Professional Development 
Gender Equity and Disability 
Prevention of Violence and Sexual Harassment 




Certification 

on nroblems with the program should be aclmowieagea, ana u sevciai 

conducteiresults should^rovided and discussed for all evaluations, not just the ones with the 
best results.) 



I certify that the information provided is accurate and conqilete. (Eigilain if necessary.) 



Submitter signature: 



REVIEWERS’ GUIDELINES 
GENDER EQUITY EXPERT PANEL 



Tking These Guidelines 

Thesejguidelines are designed to facilitate your assessment of the programs and imterials you 
review. (As defined in the OERI legislation, "program" includes produce, practices, policies, and 
research ^dings.) These guidelines parallel the Submission Guidelines in content, but have been 
reformatted, and checkoff criteria added, to make it easier for you to record your judgements and 
for others to use your review. (See the Submission Guidelines for more information on the review 
purposes and process.) 

Each submission to the Gender Equity Expert Panel contains several different kinds of materials 
that you should use in your assessment of the program: 

1 . The Cover Sheet for the submission, containing basic information about the program and 
submitter. 

2. Program background information (Part I of the submission document). 

3 ! The submitter's written responses to the review criteria (Part II of the submission 

document). These responses provide the submitter’s views of the program's strengths 
with respect to these criteria, and the evidence that the submitter feels best supports a 
recommendation of Exemplary or Promising. 

4. The program materials themselves: curricula, ^delines for using these, and other 

materials produced for use by those implementing the program. 

5. Documents containing evidence of the program’s effectiveness. These may mclude 
evaluation reports, the reactions or views of recipients of the program or other 
stakeholders, developers’ experiences with program outcomes, or other kinds of evidence 
that the program is having a positive influence on gender equity in at least one settmg. 

In reviewing a program, you should base your assessment of its effectiveness, quality, 
significance, and usefulness to others on an examination of all of these materials, although how 
carefully you will need to read each of these will depend on how relevant they are to the review 

criteria. 

You may have questions about the program that are not adequately answered by the materi^s 
included in the submission. If so, please indicate this in your review. These requests will be 
coordinated by the subpanel chairs, and the submitter will be given m oppompty to respond to 
these questions. This additional information will be taken into consideration m the subpanel s 
recommendation to the full expert panel. 

This review is confidential; reviewers will not be publicly identified, and will need to maintain the 
confidentiality of their discussions among themselves. The results of these discussions will be 
synthesized and shared with submitters and others. 

If you feel that you are not an appropriate person to review a program for which you ^ve been 
sent a submission, either because of conflict of interest or lack of expertise, or if you feel that the 
program was submitted to the wrong subpanel for review, please contact the person who sent you 
die submission (either Joseph Maxwell, at EDC, or your subpanel chair) before proceeding with 
the review. 



Thft nifflnction Between Promising_and Exemplary PrograigS 

The criteria and procedures described in these guidelines and attachments are based on the 
nroDOsed OERI regulations governing the expert panels and the standards for desig^mg 
promising and exemplary programs which can be useful to others (Dep^ent ^Edu^uon 
Regulation 34 CFR Part 701, published in the Federal Register, June 3, 1996). These draft 
regulations specify that the major distinction between promising and exen^Iary is tl^ exernpiary 
nrograms should have evidence that they work in multiple contexts and with muluple populations. 
However, this distinctibn is still under revision, and while ri is clear that the generali^mty of a 
program will be a major consideration in deciding whether it warrants "Exemplary designation, 
other factors may also be included. 

Thus the review guidelines include additional criteria for exemplary programs, relating specifically 
to evidence that the program is effective in more than one location. Submitters ana re^ewers need 
to respond to these extra criteria. If you art recommending a program as exemplary, this requires 
evidence that the program will have a positive impact in multiple sites or contexts. 



ron«;umer U«;e of the Final Recommendations an d Program Summaries 

Should a particular program be recommended as Exemplary or Promising, information ^m the 
reviews of that prom-arn, as well as the panel's discussions and the background mfo^uon on the 
program, will be synthesized into a “consumer report,” which will be fowarded to &e Secret^ of 
Miration for approval. The Department of Education will dissermnate this^omauo^n all 
nrograms that the Secretary of Education approves as either Exempl^ or Proimsmg. ims 
mfOTnation will be shared via the National Education Dissemination System to help potenual 
consumers/users make well-informed selection decisions. Feedback on all prograins reviewed 
will also be shared with the submitters so that they will obtain suggesuons for possible 
improvement of the program. This should be helpful to developers of programs that are not 
recommended, as well as for those recommended as promising. 
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REVIEW SHEET 



Appropriateness Criteria 

In order for a program to be appropriate for consideration by the panel, it must meet all three of the 
following requirements: 

It addresses the goal of advancing gender equity in or through education. (This does not 
need to be the primary goal of the program, but it must be a significant purpose or outcome.) 



It could be adopted or adapted and used by others. 

There is some evidence that the program has had a positive impact on gender equity for 
participants, smdents, or users in at least one setting or location. Tim 

necessmly have to come from a formal evaluation of the program; it could be b^ on f^back 
from participants, developers' experiences with program outcomes, or other kmds ot evidence, 
either quantitative or qualitative. 

Please check the box for each requirement if you believe that the requirement is adequ^ly met If 
you do not think that all of the appropriateness criteria are met, please contact Joseph Maxwell, at 
EDC, or your subpanel chair before proceeding with the review. 



of the Pro^am in Ter ms of the Review Criteria 

For each of the criteria listed below, there is a checkoff assessment of the extent to whi(± the 
criterion is supported by the materials provided. This checkoff hm three levels, a foi^ 
response if more information is needed; please check the appropnate box for each critenon ^»e 
is £so space for specific comments on the strengths or weaknesses of the program with respect to 

each criterion 



The criterion is met at a high level 
The criterion is met adequately 
The criterion is met weakly or not at all 

More information is needed to decide how well the criterion is met 



Individual reviewers should respond only to criteria for which they have adequae background or 
expertise. The subpanel leader may ask another panel member or adviser to review m mpect of a 
program where the assigned reviewers lack expertise, such as an interpretauon of whether it 
complies with Title IX regulations. 

Submission responses to the criteria that acknowledge both weaknesses and strengths, and that 
specify how the developers plan to address the weaknesses, should be seen as mcreasmg the 
^dMity of the response. For example, under Part 2, Section 2-K accuracy/currency , if 
submittere note that statistics on workforce participation by women have changed smce the c^nt 
version of the materials, they could say that if approved as prormsing or exemplary, they will 
update their statistics before their next printing. Or, they could mdicate where future user? could 
quickly and easily find current information. The panel may approve a prpgrarri as pronusmg if _ 
uiadequacies can be corrected in a reasonable time frame or if they would not significantly dimmish 
the program’s value for advancing gender equity. 
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1 . 



Success/Effectiveness in Promoting Gender Equity 

1 a Has the program been successful or effective in addressing problems of gencte equity in at 

least one site? (Evidence may include evaluation results, information ffona 
on a range of assessments, or other indicators that you beheve are appropnate.) If the subim^r 
claims that the program is beneficial for multiple racial, ethmc, etc. groups, and for bodi males and 
females, s/he should^vide data on program outcomes that have been disaggregated by group and 



sex. 



high level adequately weakly/not at all more information needed 



Comments; 

If you feel that the submitted program could be recommended as exemplary in advancing gender 
equity, please also respond to lb and Ic. 

lb Has the program been successful or effective in addressing problems of gender equity for 
mu'ltiple sites or widi rrmltinle populations ? The submitter should d«cuss any evidence which 
suggests that the program may ssi be beneficial for specific populauons. 

_ high level _ adequately __ weakly/not at all _ more information needed 

Comments: 

1 c. How does the program compare with similar programs in its success or effectiveness in 

advancing gender equity? 

Comments: 



Overall comments on success/effectiveness. 



2 . Quality of the program. 

2a. Is the program based on sound research and practice? 

__ high level __ adequately __ weakly/not at all 

Comments; 



more information needed 



2b Are both equity information and subject content accurate and up-to-date? 

__ high level _ adequately _ weakly/not at all _ more information needed 

Comments; 



2c. Does the program have advantages over existing alternatives? Does it complement other 
programs? 

high level adequately weakly/not at all more information needed 

Comments; 

Id Does the program prorahte equity with respect to, and is it free of bias bas^ ra, race, 

gender age culti^,^Sc origin, disability, or limited English proficiency status . (For example, 

K’l^viSStic. wide-Lging, and non-ster«.typmg portrayak of aU groups, ^d^us.ve 

respecmil language?) Does it foster high expectations for the success of all paniapanis. Does it 

comply with federal civil rights laws such as Title DC? 

weakly/not at all more information needed 



high level 

Comments: 



adequately 



2e. Is the program appropriate, engaging, and motivating for the intended audiences? 

high level adequately weakly/not at all — more information needed 

Comments: 

2f Do the materials associated with the program conform to accepted ^(tods of techmcal 
quality'^ For example, do they adhere to national computer software or audiovisual stanc^ds . Are 
diey physically durable? Are they easily usable by the intended users (mcludmg users with 

disabilities)? 

_ high level __ adequately _ weakly/not at all _ more information needed 

Comments: 



Overall comments on quality of the program; 



3. Educational Significance. 

3 a. Does this program contribute to solving or alleviating significant educational problems of 
gender equity? 

high level adequately weakly/not at all — more information needed 

Comments: 



3b. Does it address federal educational responsibilities in this area? 
__ high level _ adequately _ weakly/not at all _ 

Comments: 



more information needed 



3c. Does it contribute to increased knowledge of educational problems or issues, or improved 
strategies for teaching or learning? ^ 



high level 



adequately weakly/not at all more information needed 



Comments: 



Overall comments on educational significance: 



4. Usefulness to Others 

4a Is the program reasonable in terms of costs to potential users, in relation to the expected 
benefits and compared to other alternatives? (Costs include money, staff tune, and other requued 

resources.) 

high level adequately weakly/not at all more information needed 

Comments: 

4b Is the program easily available to potential users, and is it described clearly enough so that 
it can be adapted or adopted in new sites? Are there legal or practical constramts on its use such as 
copyrit^ht resmctions or a particularly large scope? Is it likely that odiers can unplement the 
program to obtain the same positive results as the original developer? 

_ high level _ adequately _ weakly/not at all _ more information needed 

Comments: 



Overall comments on usefulness to others: 
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fip-nera] Comments op the Proeram 

What do you see as the most important strengths and weaknesses of the program? 



■9 



What do you see as the main strengths and shortcoi^gs 
program’s recommendation as exemplary or promising? 
have strengthened the submission? 



of the evidence provided to support the 
Are there kinds of evidence that could 



What do you see as the main concerns relating to effective or ineffectwe use 

™of SnLts or wid. specific populadons? For ex^le,.has .t ^ 

single and mixed sex classes? Has it been used to benefit spec^ic popul^ons? Has it been used 

effectively in locations that have not traditionally endorsed gender equity / 



How would you compare the program with other programs having similar purposes? 



Recommendation 



Exemplary (note any inadequacies to be addressed mthin a reason^le ^e.) 

Promising (note any inadequacies to be addressed within a reasonable time.) 

panel if the Mowing changes are made, or if the Mowing additional 
niateri^s are provided: 



Do not resubmit 
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CONFIDENTIAL REVIEWER INFORMATION SHEET 



Program Tide: 

Subpanel Assignment for the Program: 

Reviewer Identification: 

Reviewer Name: ^ 

Address: 

Tel, Fax, e-mail: 

Reviewer code number 

(To be designated by subpanel chair. If possible, this code number and the date of review should 
be put as a header on each page of the review document) 

In reviewing this program, did you use any information about the program in addition to the 
materials provided by the submitter? (For exan^le, published ev^uations or other documents, 
your own experience with the prograi^ or information obtained from others who have had 
experience with the program.) Knowing if additional sources were used will be helpful to the 
panel in making a decision about recommendation and in providing feedback to the submitter. 



Reviewer Certification of No Conflict of Interest: 

I certify that I have no conflict of interest in evaluating the above program. (Please call Sue Klein at 
OERI 202-219-2038 or your subpanel chair if you aren’t sure. Do not spend your time reviewing 
unless you can sign this.) 

Signature: 

E)ate: 



Do you have any suggestions for improving the review sheet, review process, or criteria? 
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Follow-Up Papers 



Follow-Up Papers on Planning 
a System of Expert Panels 

“An Overview of OERI Efforts to Develop System to Designate and Disseminate 
Promising and Exemplary Products, Programs and Practices,” by Susan Klein 

“Expert Panels and the Dissemination of Exemplary Practices,” by John Evans 

“The System of Expert Panels: Reflections on the Future from Lessons of the Past 
Twenty Years,” by M. Christine Dwyer 

“Sharing the Best We’ve Got to Offer: Reflections on a System of Expert Panels 
to Designate and Disseminate Promising and Exemplary Products, Programs, and 
Practices,” by John Luczak 

“Dissemination and Evaluation Issues Related to the System of Expert Panels,” by 
Gary Borich 

“The Expert Panels to be Charged with Identifying Promising and Exemplary 
Educational Programs: Issues and Suggestions,” by Janet Carter 

“Other Key Issues for Planning a System of Expert Panels: Making a Complex 
System Viable, Inclusive, and Fair,” by Floraline Stevens 

“Expert Panels and Educational Reform: Some Perceptions Based on a Search for 
Promising and Exemplary Adolescent Pregnancy Programs,” by Karen Bogart 

“Reactions to the Expert Panel System as Piloted by OERI,” by Lynn Fox 

“Some Thoughts and Reflections on the OERI Expert Panel System,” by Harilyn 
Rousso 

“The Gender Equity Expert Panel: The Core Gender Equity Sub-Panel One Year 
Later,” by Marylin A. Hulme 

“Advice for Planning a System of Expert Panels,” by Mary M. Wiberg 

“Lessons to be Learned from Setting up the Gender Equity Expert Panel for 
Establishing a System of Expert Panels in Education,” by Patricia E. Ortman 



An OERI View of OERI Efforts to Develop a 
System to Design and Disseminate Promising and Exemplary Products, 

Programs and Practices 

Susan Klein' 

Office of Reform Assistance and Dissemination, Knowledge Applications Division, 

Office of Educational Research and Improvement, U.S. Department of Education 

March 19, 1997 

As with many systems under design or even those with an extensive operational history such as the 
National Diffusion Network, different individuals have different perspectives about what has been 
or should be done to help find and share the best replicable R&D models with potential users. As 
researchers we want perspectives to be informed by good analytic insights which include learning 
from related activities. We also understand the value of consistent sets of principles and 
assumptions about what is needed and how to accomplish the goals. 

This paper describes my views about OERI’s leadership role in developing a System of Expert 
Panels to identify and share promising and exemplary products, programs and practices. My ideas 
are based on what is covered by the 1994 OERI reauthorization legislation^ as well as research that 
I and my colleagues have been doing since the mid 1970s on how federal offices learn about and 
share the best (Klein, 1993, 1996; Klein & Gwaltney, 1991; LaFollette, 1992). These ideas are also 
described in an article that should be published in the Educational Researcher soon on “A System 
of Expert Panels and Design Competitions: Complementary Federal Approaches to Find, Develop 
and Share Promising and Exemplary Products and Programs.” This article responds to Bob Slavin’s 
article on “Design Competitions: A Proposal for a New Federal Role in Educational Research and 
Development for the Jan/ Feb. 1997 Educational Researcher. 

Background and Purposes: Under its 1994 Reauthorization (Title IX of Goals 2000: Educate 
America Act), OERI has been given responsibility for: 

• developing standards to designate promising and exemplary products, programs and 
practices. Draft regulations for these standards published June 3, 1996 were revised in 
response to the public comment and will be published in draft again for additional public 



' This paper is prepared for the Annual Meeting of the American Educational Research Association in Chicago, 
Symposium 5.26 “Learning from Consumer-Oriented Review Efforts to Designate Promising and Exemplary Products, 
Programs and Practices” on March 24, 1997. It is based on a longer paper “A System of Expert Panels and Design 
Competitions; Complementary Federal Approaches to Find, Develop and Share Promising and Exemplary Products and 
Programs,” that has been cleared by the Department of Education for publication in the Educational Researcher. This shorter 
paper and the presentation upon which it is to be based are intended to promote the exchange of ideas among researchers and 
policy makers. The views are those of the author and no official support by the U.S. Department of Education is intended or 
should be inferred. 

^ I was fortunate in being able to advise Congressman Major Owens and his Subcommittee staff as they prepared this 
legislation. 
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Secretary of Education. Two pilot panels in the areas of mathematics and science, 
and gender equity were appointed in 1996. 

• coordinating dissemination activities and programs not just within OERI, but with other parts 

of the Department, other agencies concerned with education, and with associations and other 
levels of government, such as state education agencies and local school districts. 

The System of Expert Panels should enable the federal government to: 

1) help the public leam about the comparative advantages of what exists among the many available 
replicable R&D-based products, programs, practices and policies^ based on an understanding of their 
relative merits'* *; 

2) help federal and other funders and producers of R&D-based resources maximiTp. their investments 
by providing support for further evaluation, improvement and dissemination of existing promising 
and exemplary R&D-based resources and by identifying gaps which may help indicate areas where 
new products and programs are needed. Other expected benefits include providing practical reasons 
for educators (and in some cases students) to improve their evaluation skills related to the use of 
evaluation information in making decisions on instructional tools such as programs or products. 
Participants in System activities will also be able to contribute to the development of new methods 
to combine evaluation and dissemination functions to better serve education consumers. 

This System is congruent with the Department of Education’s strategic plan (1994) to better 
serve its customers, but it is not limited to reviewing models that have been developed with 
previous federal funding. It builds on, but differs from, the Joint Dissemination Review Panel 
and its successor the Program Effectiveness Panel used by the Department’s National Diffusion 
Network (NDN) which is no longer funded.^ 



3 r 

These replicable R&D-based resources: 1) can be used outside of the original development site, 
2) are based on principles from educational research and 3) have some evaluation evidence on their 
positive impact. In the. June 3, 1996 draft standards and in this paper, “program” will often be used to 
refer to all these R&D-based models or resources. The OERI legislation includes research findings in this 
list, but instead of having expert panels designate promising or exemplary research studies, syntheses and 
interpretive papers or informational videos, it is most likely that research findings or principles will be used 
indirectly by including them in the criteria established by each panel. 

“* Merit is determined by expert panel reviews using criteria under the four categories in the OERI 
standards for designating promising and exemplary programs. These categories are: evidence of 
effectiveness/ success, quality, educational significance, and usefulness to others. (See the sidebar.) 

* The National Diffusion Network (NDN) established in 1974 and last funded in 1995 provided a 
systematic procedure to identify and disseminate programs that had evidence of effectiveness. The System 
of Expert Panels builds on the NDN focus on reviewing submissions to judge effectiveness of replicable 
programs. But it goes beyond NDN in many ways, such as including three additional criteria categories 
and the deliberate use of experts, criteria and evidence appropriate for specific topic areas. NDN was a 
self-contained general purpose review and dissemination system with a review panel, ED funding of many 



Structure and Operational Principles: 

Structurally this Federally Led Nationwide System Includes More Than Expert Panels. 

It is important to develop a well understood and articulated nationwide system that has clear 
connections to other complementary systems, such as the National Education Dissemination System 
(NEDS) and the even more loosely configured R&D production system. Ideally, this System would 
be logical, inclusive and dynamic. As explained later, the federal role would be facilitative and 
collaborative, rather than regulatory. Experience v^th the development of the Gender Equity Expert 
panel shows that thinking in new ways is a challenge for many of us on even simple issues such as 
public acknowledgment about who has submitted what. The community of panel members and their 
advisers need to decide if they want all submissions to be known publicly and applauded and helped 
by the coimnunity or if they want them to be confidential until positive decisions are made by the 
Secretary of Education about their promising or exemplary status. 

The System of Expert Panels should be more than a collection of individual expert panels. The 
“Wheel Diagram of the Key Elements of the System of Expert Panels” shows that the System would 
be led and coordinated from the inside hub circle by a Federal Coordination Unit (FCU) of experts 
from various Department of Education Offices and other agencies involved in education activities 
to identify and share the best from R&D. This FCU would include agency representatives such as 
federal staff liaisons with current or potential expert panels. 

The middle circle connects the individual expert panels (P) in a comprehensive array of topic areas. 
Aside from the pilot Expert Panel in Mathematics and Science Education, other curriculum areas 
such as reading may be included. Among others with an equity focus, special education may join 
the Gender Equity Expert Panel. Other expert panels may focus on topic areas such as school-wide 
models, safe and drug-free schools, uses of education technology, and teacher education. Where 
feasible, each topic focused expert panel would have liaisons from federal offices who would be in 
a position to use money and knowledge from their programs to contribute to and build on panel 
work. These federal liaisons would also advise on the selection of panel members who would be 
approved by their leaders. This circle also contains a standing panel and a democratic governance 
and coordination structure for the expert panels who would represent their topic-focused 
constituency groups. 



System of Expert Panels builds on the NDN focus on reviewing submissions to judge effectiveness of 
replicable programs. But it goes beyond NDN in many ways, such as including three additional criteria 
categories and the deliberate use of experts, criteria and evidence appropriate for specific topic areas. 
NDN was a self-contained general purpose review and dissemination system with a review panel, ED 
funding of many of those approved by the panel as Developers/Demonstrators (to provide 
implementation assistance to adopters) and State Facilitators (to encourage the use of all approved 
programs). 
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Wheel Diagram of the Key Elements of 
the System of Expert Panels 
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The outside circle represents the consumer-oriented disse minati on and evaluation providers (D&E). 
But, unlike the other parts of the diagram, this outer circle p rimar ily would be composed of existing 
dissemination and evaluation entities such as ERIC, tec hni cal assistance centers. Regional 
Educational Laboratories, organizations that provide third-party evaluations. National R&D Centers, 
publishers, participants in design competitions, and organizations managing ED and other related 
World Wide Web sites. Some of these D&E providers would specialize in specific topic areas and 
some in multiple topic areas. In some cases, national topic specific disse minati on and evaluation 
organizations would be prime candidates to serve as support contractors for an expert panel in their 
topic area. For example, the Women’s Educational Equity Act (WEEA) Equity Resource Center is 
the support contractor for the Gender Equity Expert Panel. If needed, new topic specific 
dissemination and evaluation services could be created to support the work of expert panels in 
specific topic areas. It is likely that most D&E providers would also have key roles in other aspects 
of the NEDS and the more loosely configured R&D production system. Like the middle circle, 
participants in this outer circle would be linked across Panel topics by a governance structure® and 
by joint use of Special D&E Providers. Functions of topic specific and multi-topic Special D&E 
Providers would include disseminating promising and exemplary programs, obtaining and sharing 
systematic user feedback, reviews and evaluation reports on promising and exemplary programs in 
multiple topic areas and sharing information about opportunities to participate in the third-party 
evaluations of promising programs. In a commissioned paper, Patricia Campbell (1994) described 
initial ideas on how a D&E provider could obtain, synthesize and share teacher and student 
evaluation feedback on promising and exemplary programs. 

The spokes of the wheel show that much of the coordination of the work and contacts with the users 
would be organized in relation to each of the topic focused expert panels. However, there would also 
be many other cormections since some dissemination and evaluation stmctures cover multiple topic 
areas and since promising and exemplary programs (i.e., gender equity in mathematics, science and 
technology) may be appropriately disseminated under the auspices of more than one topic area. 

Principle 1. The Federal Leadership Role Should be Facilitative, The federal gove rnme nt will play 
a facilitative leadership, partnership, community building role, more than a regulatory role. This new 
type of collaborative leadership with the field can be developed by establishing expert panels in a 
wide variety of topic or special interest areas and sustained over many years with relatively modest 
federal investments because the topic-focused constituency groups would volunteer review and other 
services, much as they do now in reviewing journal articles. In addition to strengthening support for 
the agency from its concerned constituencies, this approach should help these topic focused 
communities develop a systematic process to share what works and identify gaps. The current panels 
are succeeding in this in two different ways. The Gender Equity Expert Panel has created an 
advisory group of over 100 experts who are willing to help the Expert Panel. And many of the 
members of the Mathematics and Science Expert Panel are leaders of key constituency groups in 
their area. This closer cormection with constituency groups who have expertise in particular topic 



® This governance structure may be part of NEDS or it may be created specifically for D&E 
providers with ties to the expert panels. 
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areas should also help make agency supported work in the area such as design competitions, and 
technical assistance activities more relevant to constituent needs.’ 

A facilitative federal coordination function is needed to help all components of the System learn 
from each other and work in complementary ways where there are natural overlaps in 
responsibilities. More specifically, it will: 1) provide leadership in coordinating (and where feasible, 
funding) all components of the Wheel Diagram, 2) develop systematic evaluation and dissemination 
methodologies and consistent decision rules and definitions for system activities so that the public 
will understand distinctions between “promising” and “exemplary” programs, and 3) develop 
incentives to make the System work. 

1) Coordination of components of the Wheel Diagram requires flexibility to take advantage of the 
interests of various topic focused federal offices and constituency groups. In doing so, it is assumed 
that federal programs in and outside of OERI would participate in this System, but that each would 
have somewhat unique activities to take advantage of their own legislative responsibilities. For 
example, in mathematics and science education, NSF might be able to support a D&E provider to 
manage comparative third-party evaluations of promising programs with similar purposes or they 
might fund a D&E provider to manage a design competition (as suggested by Robert Slavin, 1997) 
to fill a specific gap in the availability of exemplary replicable programs. ED might be able to 
support state professional development programs that choose promising and exemplary products and 
programs recommended by the Mathematics and Science Expert Panel or the Gender Equity Expert 
Panel. 

2) Development of common evaluation and dissemination methods and definitions has started. The 
draft OERI standards for designating promising and exemplary, programs have common criteria 
categories: 

- evidence of effectiveness/success (from self and third-party evaluations of performance) 

- quality (panels judge this by reviewing program materials, perhaps by observing the 
program in action. They may use criteria such as: congruency with research, accurate & 
up-to date content, promotes equity and is free of bias based on race, gender, age, culture, 
ethnic origin, disability or limited English proficiency, 

- educational significance (addresses important educational issues, has advantages over other 
programs with similar purposes) 

- usefulness to others (See Summary of OERI Standards and Criteria). 

3) Development of major System incentives revolves around consequences for receiving designations 



’ Constituency groups as represented by associations often recognize excellence by giving awards 
to individuals in their field or they help establish standards and credentialling systems which focus on 
judging the quality of individuals or organizations in meeting standards of the profession. They have 
established activities to recognize promising and exemplary programs, but often find it difficult to sustain 
this type of review effort without external support or clear cormections to continued evaluation and 
dissemination efforts. The Department of Education has had similar experiences with a wide range of 
recognition and review activities. 



of promising or exemplary. One suggestion is to provide funds for improving and evaluating 
programs that are designated “promising”. This support would be geared to helping them qualify 
for future exemplary designations. Based on this understanding, a promising designation would 
encourage only cautious adaptation or adoption, during which time the program would undergo 
careful evaluation. Users would be encouraged to select exemplary rather than promising programs 
unless no exemplary programs meet their need, or unless they wish to participate in the field testing 
of promising programs. The key purpose for designating programs as exemplary is to increase the 
use of what works best. In addition to developing working relationships with specific D&E 
providers, the Expert Panels would work with others to publicize information on exemplary 
programs through the World Wide Web, TV and print media to help consumers choose from a 
variety of particularly worthwhile options. 

The active partnership roles for the topic-focused constituency groups will need to build on their 
strengths, resources and interests. For example, the Gender Equity Expert Panel has formed six 
subpanels to better cormect with specific interest groups and the Mathematics and Science Expert 
Panel members have many contacts with mathematics and science educators they can tap through 
state affiliates of their national associations. 

Principle 2. There should he a combined focus on evaluation and dissemination which I call 
Consumer-Oriented Evaluation. This focus should be particularly appealing to educators who 
know they can do a better job if they are able to choose and use effective tools or instructional 
programs. This concept focuses on obtaining and disseminating descriptive and evaluative 
information designed to help consumers make decisions about what products or programs will be 
best for them. In addition to educators, consumers include the entire public (ranging from policy 
makers to students) who might be interested in learning about the merit of an education product, 
program or practice. Since many of these R&D-based tools have similar purposes, they will want 
to know their comparative merit on a range of criteria that are likely to be important for their own 
decision-making. The OERI standards categories as described in the sidebar are intended to provide 
a framework for topic-focused criteria and for research-based criteria selected by experts. Ideally, 
this System of Expert Panels should use D&E providers to develop consumer reports® for specific 
topic areas where there are Expert Panels. These consumer reports could be supplemented by 
descriptions and summary reviews of individual promising and exemplary programs which a 
potential user could retrieve via a computer/Intemet search. Consumers should be able to use either 
or both of these information sources so they can do their own side-by side comparisons and malcp 
informed selection decisions. Based on past experience from Educational Products Information 
Exchange (EPIE) (Komoski, 1989) and others, it is unlikely that education consumers wUl want to 
pay for this information. So it is assumed that the government, foundations and associations will 
need to pay for D&E providers to do this public domain work. When all System components are 



g 

Consumer Reports from the Consumers Union could be one model. Each report would contain 
an overview describing the programs and key issues related to their merit, a comparison chart and 
individual program summaries. 



operating, the evaluation results that would be included in this consumer information would not end 
with initial expert panel decisions. The panels and their support contractors would continue to up- 
date this information based on work of the D&E Providers described in the outside wheel of the 
System Diagram. For example, a Special D&E Provider would collect information from users of 
promising and exemplary programs, thus making sure that users play a continual role in helping each 
other learn what works for whom. It is assumed that this new feedback function and many of the 
nationwide operations of this System will become increasingly feasible as more educators use 
computers and Internet to communicate. 

Principle 3. This System should help create a deliberate, effective and continuous federal 
approach to finding and sharing the best and identifying gaps which merit R&D support. Many 
federal education programs fund “demonstration” projects which generally allow for the 
development of a creative program in one site. But when federal funding ends the program may also 
end in the original site and there are few opportunities to learn if it merits continued support for 
additional revision, evaluation or dissemination to others. Similarly, federal offices rarely provide 
this type of additional support for meritorious programs not developed with federal funds. Thus, the 
System of Expert Panels is intended to carefully identify the gaps and opportunities and may help 
target subsequent federal funding to replicable programs with the greatest chance of helping students 
receive a better education and show improved performance. While most Expert Panels will be 
designed to cover the broad interests of their constituency groups and to last over a number of years, 
it is also possible for more short-term specific focused expert panels to be established and funded 
by a federal office or for a broader panel to limit their search for solutions to priority areas during 
specific time periods (Datta & Scriven, 1997). 

Principle 4. National dissemination efforts should find multiple ways to encourage selection and 
use of the best education R&D has to offer. 

This System is not based only on recognition and awards for excellence. Instead it focuses on 
learning about or developing what is likely to be useful to others. The assumption is that most users 
will find the funds for implementation especially of exemplary models or designs in late Stages of 
development. If programs are designated as exemplary by the System of Expert Panels the federal 
government could encourage recipients of Title I or other federal funds to choose from among the 
exemplary programs or provide technical assistance or other support for their implementation. 
Incentives could also be provided to collect evaluation evidence on promising and exemplary 
programs to learn more about how they work in different situations. This evidence could be 
collected, analyzed and shared by a D&E provider. 

Principle 5. All participants will increase their commitment to using research, development and 
evaluation to learn what works well to inform and foster revision (adaptation) and replication 
and thus improve education. Many recognize the need to help schools obtain good evidence on the 
effectiveness of replicable models so they will have justification for choosing among them or 
sticking with their current practices. 
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Principle 6. The System of Expert Panels is very ambitious and likely to face challenges such as: 

• Obtaining major financial support for individual panels and for related D&E providers. 

• The replication of what works focus may also be opposed by educators who believe that 
replication of model programs and products is not a feasible strategy for educational 
improvement because of the need for substantial adaptation’ or because they think educators 
must develop their own approaches based on their interpretations of research findings and 
their own insights and experiences. 

• Technical and procedural challenges related to deciding on the promising or exemplary 
status of submissions to the expert panels are substantial. Examples include difficulty 
in agreeing on desired performance indicators, frequent lack of information on adequacy 
of implementation as well as performance outcomes, and practical difficulties in making 
sure that evidence collection and panel submissions will be “the whole truth and nothing 
but the truth”. Although the System is designed to judge the program, not the developer, 
it is difficult to separate developers from their programs especially when the evaluations 
are likely to be dependent on the cooperation of the developers and thus they would not be 
anonymous. 

Conclusion: 

The other presentations in this session and the experiences of the pilot expert panels should help us 
learn how to build on the work of others to make this System a basis for the U.S. Department of 
Education to develop a systematic integrated leadership approach toward working with R&D and 
practitioner experts in a wide variety of topic areas. The symposium 16.23 tomorrow afternoon on 
the Gender Equity Expert Panel will show how important it is for experts in R&D to work together 
to learn what works best in their areas of special interest. (I thank both the AERA Special Interest 
Group: Research Utilization and the AERA Women’s Committee and the Special Interest Group: 
Research on Women in Education for sponsoring these sessions.) 
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OERI STANDARDS AND CRITERIA FOR EXPERT PANELS TO USE TO DISTINGUISH BETWEEN PROMISING AND 
EXEMPLARY PROGRAMS— summary based on 2/1 1/97 draft regulations revised according to public comment and suggestions 
from the OERI Board at their Jan. 31, 1997 Meeting in Washington, DC. A subsequent version of these draft regulations is to be 
published in the Federal Register in the spring of 1997 for public comment. 

A panel may recommend to the Secretary that a program be designated as promising if the panel determines that the program is 
strong on each of the four categories of standards (Evidence of Success/Effectiveness, Quality of the program. Educational 
significance and Usefulness to others). A panel may recommend to the Secretary that a program be designated as exemplary if the 
panel determines that the program is excellent on each of the four categories of standards. The decision firework for 
determinations of strong and excellent on evidence of success/effectiveness is intended to be the same for all panels. However, on 
the other three standards categories of quality, significance and usefulness to others, each panel may add to the core criteria and 
establish its own decision framework for designation as promising and exemplary. It is expected that the panels will require a strong 
judgement on almost all criteria within each of these three categories for a designation as exemplary. For a program to be designated 
exemplary any weaknesses must be minimal and easily corrected. 

(a) Evidence of Effectiveness/Success. 

To be judged strong in the category of evidence of effectiveness all of the following criteria must be met. A program must: 

• have defensible overall evidence supporting claims of worthwhile performance results (without substantial harmful results) at 
one or more sites (without failing at a large number of other sites) 

• have logical or other evidence of adaptability or transportability to other sites. 

The combination of this positive evidence of effectiveness and potential replicability creates a significant probability that the program 
will eventfiilly be able to provide evidence to support claims of exemplary meritorious results as defined in the following section. 

To be judged excellent under the category of evidence of success/effectiveness all of the following criteria must be met by convincing 
evidence that very important claims of positive results (or performance outcomes) can be: 

• reasonably attributed to the program, and 

• that evidence to support these claims of worthwhile results was sustained in multiple site replications within the past few 
years (without failing at a large number of other sites or being accompanied by harmful results). 

(h) Quality. The panels will make their judgments about quality by reviewing the program materials and determining the extent to 
which the program: 

(1) is congruent with sound research and practice 

(2) incorporates accurate and up-to-date information/content 

(3) promotes equity and is free of bias based on race, gender, age, culture, ethnic origin, disability, or limited English 
proficiency 

(4) is appropriate, engaging, and motivating for the intended audiences 

(5) contains materials that conform to accepted standards of technical product quality. 

(c) Education Significance. The panels will use their expertise in the area to determine the extent to which the program: 

(1) addresses an important education issue, challenge or problem 

(2) has advantages over other programs with similar purposes 

(d) Usefulness to Others. The panels will make these judgements by using their knowledge of what is valued by educator and 
student users in determining the extent to which the program: 

(1) is reasonable in terms of costs to potential users in relation to expected benefits 

(2) is or can be made easily available to potential users 

(3) can be readily adopted or easily adapted in new locations 

(4) Can be used in conjunction with other programs if appropriate 



A-12 




G7 



Expert Panels and the Dissemination of Exemplary Practices 



John W. Evans, Ph.D. 

Rev. May 15, 1997 

This paper responds specifically to OERI's request for advice on the formation of expert panels, as 
called for in the Goals 2000: Educate America Act, but it also compares this new effort to previous 
ones and places them in the larger context of federal education dissemination policies and programs. 



It is unquestionably a good thing that a principal piece of legislation governing the operation of the 
Department of Education calls for expert panels to judge the effectiveness of programs the 
Department proposes to disseminate. Of course, the idea of such panels is not new, so it is 
instructive in understanding how important they are and how they can effectively operate to look at 
the history of their predecessors. 

• Previous Dissemination Review Efforts 

In the 1970s, it was common practice for the Department's' K-12 programs — elementary and 
secondary, vocational, handicapped, and bilingual — ^to engage in a broad range of dissemination 
activities: pamphlets, official recommendations, demonstrations, project awards, education fairs, 
etc. However, when the question was raised as to what evidence existed that these federally 
recommended projects were effective in improving reading, enhancing skills, raising motivation, 
lowering drop out rates, etc., the answers from the program staffs were not reassuring. It was said, 
with extensive use of anecdotes, that these programs were being disseminated because they looked 
impressive on site visits, students and teachers favored them, observers had judged them to be 
educationally sound, other schools had decided to adopt them, and so on. 

In a discussion of the Department's dissemination policies and procedures with then Commissioner 
Sydney Marland and his successor Terrel Bell, several of us who were responsible for evaluation 
and research in the Department pointed out that placing a federal stamp of approval on programs 
carried with it a large responsibility. We were advising schools to set aside existing programs, spend 
money to adopt the ones we were recommending, and rightfully expect educational improvements 
from them. If these programs, most of which had no objective evidence of effectiveness, did not 
produce the results they promised, the schools would be misled, the dissemination process would 
become suspect, and the Department's credibility would be damaged. 

Out of these discussions emerged the initial Dissemination Review Panel, which later became the 



'At that time, the Department of Education was, of course, the U.S. Office of Education, part of HEW. 
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Institute of Education (NIE) with its own dissemination responsibilities. A formal policy was issued 
that required all disseminations be approved by the JDRP. 

The JDRP was staffed with Department people who had research and evaluation backgrounds. It 
developed criteria that emphasized objective evidence of effectiveness, and it set up procedures 
aimed at simplicity — ^periodic meetings to review a standardized submission form and a vote to 
accept or reject the project for dissemination. During the years of its operation, the JDRP typically 
approved about 50% of the submissions it received. Since the panel utilized existing Department 
people who served on it in addition to their regular duties, it functioned without cost, except for a 
part-time person who handled the a dminis trative work. 

Not surprisingly, the JDRP was not immediately popular with the programs. It was now necessary 
for them to present strong evidence of effectiveness to the JDRP for projects and programs they 
wished to disseminate. As a consequence, their past freedom to disseminate whatever they wished 
was sharply curtailed. 

But the JDRP came to be accepted, and its reputation grew. JDRP approval became a symbol of 
quality that program staffers enjoyed using in their dissemination efforts. Several volumes were 
published of projects the JDRP approved.^ As JDRP review became the approval mechanism for 
projects in the Department's National Diffusion Network (NDN), states formed their own versions 
of the panel. All of this had the important effect of expanding the basic notion that professed 
excellence should rest on hard evidence of effectiveness. 

The JDRP was succeeded by the Program Effectiveness Panel (PEP), which used similar criteria, 
people, and procedures, except that it rehed mainly on mail panels of external experts not on the 
Department staff. 

So what does this history tell us about the current efforts to establish a new set of expert review 
panels? First, the continued reemergence of such panels — now with legislative 
authorization — vindicates that the basic idea of requiring responsible federal dissemination efforts to 
rest on objective evidence of effectiveness is compelling. 

But the history also raises puzzling questions about the repeated demise of these panels and the 
dissemination programs they serviced, like the NDN. As Santayana said, "Those who cannot 
remember history are condemned to repeat it." To those familiar with the Department's past efforts 
at dissemination review panels, the new proposals have the uncomfortable Yogi Berra feeling of 
"deja vu all over again." The past panels were successful — some would say exemplary — ^in 
establishing criteria and procedures for identifying effective projects that warranted national 
dissemination. Yet, for a variety of reasons related to budgets, reorganizations, and pohtics — ^none 



Educational Programs that Work: The Catalogue of the National Diffusion Network," 22nd Edition, 
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of which were clear or convincing — these valuable and productive activities were discontinued. 
Now, instead of being in a position to build on established efforts that are integrated with other 
Department programs and respected by state and local educational agencies, the Department appears 
to be engaged, once again, in reinventing the wheel, albeit this time with legislative authorization. 

While it is important to understand this history (hopefully to minimiz e additional repetitions of it), 
it is not useful to wallow in it. The previous review panels have been dismantled, new legislatively 
authorized versions are now on the table, and the task at hand is to make them work. 

To get, then, to the assignment of providing some assessment and advice to OERI about the 
implementation of the new expert panels, I offer the following observations and recommendations: 

• The Positive Features of the New Expert Panels 

• As I said at the outset, the basic concept of establishing expert panels that will assess the 
effectiveness of programs proposed for dissemination primarily on the basis of objective 
evidence is excellent, and it is good to have this standard codified in legislation. The 
summary handed out at the AERA meeting stated that, "The purpose of the System of Expert 
Panels is to ensure that the programs disseminated by the Department are high-quality, 
research-based programs that have provided evidence indicating that they have improved 
teaching and/or learning." This is an excellent summary statement because it makes clear 
that approved projects (1) should be supported by solid evidence of effectiveness, and (2) 
they should demonstrate improvements in teaching and learning. 

But it is unclear whether the Department is prepared to make it a requirement that all 
dissemination of projects described as exemplary, effective, or promising must receive the 
approval of an expert panel, thereby having the panels function as the Department's quality 
control mechamsm in the area of dissemination. If no such requirement exists, the original 
problem addressed by the JDRP — ^promiscuous dissemination under a federal stamp of 
approval of projects lacking evidence of effectiveness — can be expected to continue, with 
the resulting loss of federal credibility when they do not produce the promised results. 

• The proposed professional qualifications of expert panel members appear to be appropriate, 
but since the ultimate criterion of approval is evidence of effectiveness, professional tr ainin g 
and experience in research and evaluation should be the sine qua non of panel qualification. 
It may seem that "expertise in a specific educational area" is a reasonable requirement, but 
care must be taken to prevent a situation in which panel members function as advocates for 
their constituents' projects. 

• It is clear that considerable attention has been paid to the distinction between exemplary and 
promising, and I believe it can be made to work, although there are difficulties and threats 
to be concerned about. This same distinction was proposed for the JDRP, but it was not 
adopted because of the concern that once a stamp of federal approval was given, the 
distinction between exemplary and promising would be lost. We were not able to work out 



a clear evidentiary distinction between the two, and we were concerned that projects unable 
to achieve the level of evaluation evidence required for an exemplary designation would 
flock to the promising level on the assumption that "once it's approved, it's approved, and 
nobody will quibble over the difference." 

However, though I still have these concerns, I am impressed with the efforts that have been 
made to make this distinction work and thereby legitimately expand the number of projects 
that can be approved for dissemination. The key, I believe, is to rigorously hold to the 
proposed requirement that projects designated as promising must have defensible evidence 
of effective performance in one or more sites without failing at a large number of other sites. 
In addition, since the promising designation calls for more definitive evaluation, that could 
act as a general encouragement for better evaluations, and it could guide future investments 
in evaluation. 

Concerns About the New System 

Although I believe the distinction between exemplary and promising progr ams can be maHp 
to work, there were repeated comments made by the members of our subgroup at the AERA 
session that it will be a difficult distinction to consistently make in practice, and that 
numerous clarifying examples should be developed. 

On the matter of the four criteria (effectiveness, quality, educational significance, and 
usefulness), the language which has been developed makes clear that all projects, including 
those at the promising level, must have acceptable evidence of effectiveness to be approved. 
However, the functioning of the panels will need to be monitored closely to insure that the 
effectiveness criterion is, in practice, preeminent. 

This issue was confronted in the development of policy for the JDRP. We had seen a well 
intentioned but unfortunate effort made by the elementary and secondary programs to 
develop a set of criteria similar to the four now being proposed. The fatal problem with that 
effort was that it was possible to receive approval with a total score across all the criteria in 
which the score for effectiveness was zero. We finally decided to intentionally limi t the 
review and the basis for approval to evidence of effectiveness. We felt that other criteria 
such as quality and usefulness were specific to the adopting site and involved complex 
educational judgments that were beyond the knowledge of a panel constructed primarily to 
judge evidence of effectiveness. 

Such a limitation would be my recommendation for the new expert panels, but it appears that 
decision has been cast in the direction of including the other criteria. If that is indeed the 
case, the need to insure that effectiveness is the necessary criterion is all the mote important. 

A major concern I have about the proposed expert panel system is its informal quality. The 
image that emerges is of a loose, casual set of panels that are convened occasionally. How 
are the members to be selected, compensated, supervised, and evaluated? How will the 



panels function? Under whose direction? Will there be actual meetings where discussions 
can occur between the submitter and the panel and among the panel members, or will 
determinations be made by mail, as in the case of the PEP? 

The lack of a formal system carries the message that the Department is not serious about this 
effort, and that it proposes to discharge this function as casually and cheaply as possible 
without any major pohcy, organizational, or budgetary changes. 

If the system is to function productively and credibly, it must have, in the best Weberian 
sense, some bureaucratic reality. As it is. now, there is, in Gertrude Stein's terms "no there, 
there." 

The most serious concern I have about the new expert panels is that they are not part of a 
larger. Department-wide system that elevates dissemination to a high priority and provides 
the necessary prominence and resources for it to succeed. Instead, the system appears to be 
a fragmented patchwork that rests on unrealistic assumptions about the involvement and 
contributions of other organizations and individuals. 

The proposed system would apparently depend on major contributions from ERIC, the 
Regional Educational Laboratories, the National R«feD Centers, other organizations, and 
volunteering individuals. Of course, these organizations have important contributions to 
make to educational dissemination, but it is not realistic to imagine that collectively they can 
be "coordinated" into forming the corpus of a new federal dissemination system. They have 
other missions, priorities, projects, and commitments. It is fanciful to assume that they can 
or will abandon their current work and redirect their efforts to dissemination activities 
directed from Washington. 

Moreover, apart from the assumed contributions of these organizations, there is no provision 
in the proposed system for a major capability to assist schools in implementing approved 
projects, as the NDN previously did. It is axiomatic in the dissemination field that in many 
schools such assistance is cmcial to the successful adoption of new practices. 

Efforts to create new programs and initiatives by re-labeling existing ones are not uncommon 
in the Federal Government. They provide the impression of accomplishment, but eventually 
their illusory quality is exposed. 

This is unfortunate and unnecessary. Given the limited and specialized role the Federal 
Government plays in the American educational system, rigorously determining what 
practices are effective and providing the support necessary to get them into widespread use 

ought to be one of the Department's primary functions, and the supporting staff and budgets 
should reflect that. 

Historically, most pieces of the Department's authorizing legislation have called, in varying 

B-5 




< • ^ 



72 



fashions, for the dissemination of exemplary practices to state and local agencies. The Goals 
2000: Educate America Act repeatedly and explicitly instructs the Department to make. 
dissemination a top priority and includes an entire section (Part D) on establishing a National 
Education Dissemination System. 

Thus, the Department has the opportunity and the mandate to make dissemination not just 
one of the divisions in its research department but one of its defining priorities and do minan t 
activities. Such a major change would also be responsive to the many state and local 
educational reform efforts. In addition, a major thrust to expand the effort to identify 
effective practices and assist in their implementation nationwide would likely have the 
important side effect of improving the Department's (and particularly OERI's) tarnished 
reputation in the Congress. 

Summary 

This brief paper does not touch on many of the unresolved issues in the field of educational 
dissemination, such as the questions of adoption vs. adaptation, the differences between small, 
classroom projects vs. large, school-system reforms, and others. It responds to OERI's request for 
advice on structuring and implementing a new system of expert panels to judge projects proposed 
for national dissemination. 

Such panels are an essential component of any responsible disse minati on system. They provide the 
indispensable basis for objectively determining whether projects proposed for national dissemination 
have credible evidence of effectiveness. If the panels are used by all Department progr ams , they will 
constitute a quality control mechanism for the Department's dissemination activities and provide a 
high level of assurance to state and local educators that federally recommended projects will likely 
produce the results they promise. 

The qualifications for panel membership are generally satisfactory, but care must be taken to insure 
that evidence of effectiveness is the preeminent criterion of approval, that panel members have 
evaluation and research expertise, and that they do not function as advocates for the projects of their 
content-area constituents. 

The distinction between promising and exemplary is difficult to make, both concepmally and in 
practice. But the efforts that have been made to do this are impressive, and I think the distinction 
can be made to work. It is important that projects designated as promising fully satisfy the 
requirement of evidence of effective performance in one or more sites without failin g at a large 
number of other sites. It will also be necessary that clarifying examples of this distinction be 
provided to panel members. 

Despite these promising features of the proposed expert panel system, there are major problems to 
be confronted and resolved if the panels are to be more than a cosmetic change to the present review 
system, a system that is largely non-existent following the discontinuation of the JDRP, the PEP, and 
theNDN. 
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First, the loose and informal character of the proposed panel arrangement does not inspire confidence 
that it can fimction as the heart of an effective and credible national dissemination system. A clearer 
and stronger organizational form is needed, along with the necessary resources, guidelines, and 
oversight. 

Second, my most serious concern is that the new expert panels c ann ot function productively unless 
they are part of a larger. Department-wide system that elevates disse minati on to a high priority and 
provides the necessary prominence and resomces for it to succeed. Instead of making dissemination 
one of its central missions — a mission that would be especially appropriate for the Federal 
Government's limited, supporting role in the American educational system and its commitment to 
educational reform — ^the Department's response to the new legislation appears to be largely one that 
re-labels existing activities and relies on the imagined contributions of other organizations and 
individuals. Under these conditions, it is hard to be optimistic that the new panels, by the ms elves, 
can make a significant contribution to the national dissemination of exemplary educational practice. 
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The System of Expert Panels: Reflections on The Future From Lessons of The 

Past Twenty Years 

M. Christine Dwyer 
RMC Research 
May 1997 

This paper is written from the perspective of one who was an "insider" with the processes formerly 
used by the Department of Education to identify, validate, and disseminate exemplary programs. 
I was engaged in providing technical assistance to the Program Effectiveness Panel (PEP) and the 
National Diffusion Network (NDN) starting in 1983 through its conclusion in September 1996 
(referred to in this paper as "the former system"). During that period of time, I was responsible along 
with my RMC Research colleagues for both making suggestions for changes, including the 
development of the PEP criteria and guidelines, and then helping to implement those changes in 
operational practices. To follow the development of thinkin g around the proposed new system for 
the validation and dissemination of best practices, I have read the many papers commissioned to lay 
the groundwork for a new system, considered the reported experiences of the two initial expert 
panels, and reviewed the proposed description of the expert panel system. 

My task for this paper is to consider the proposed system in light of the experiences and lessons 
learned through previous federal efforts since the expert panel system is designed to build on but 
differ from past structures. The paper discusses several persistent misconceptions or faulty 
assumptions about the former system. To the extent that some of the ideas in the proposed expert 
panel system have been influenced by those misconceptions, the new system may not realize its 
potential. Some design features may not actually be improvements over former approaches or the 
new design may not have addressed yet substantial problems that hindered the former system. My 
observations are based on direct experiences with PEP and NDN and also the Blue Ribbon Schools 
and the national Title I Recognition programs. 

There are four common misconceptions that pervade the dialogue around the former system that 
have implications for the design of the expert panel system. Those misconceptions are in four areas: 

Identification of programs: It is a myth that programs were primarily on their own to 
find their way into the system. 

Evaluation requirements: It is not true that program evaluations had to be 
experimental or quasi-experimental in design nor that test scores were required as 
evidence. 

Value of projects: The system did validate and disseminate some important and 
successful projects. 
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• Philosophy of change: The NDN philosophy of educational reform evolved over 

time from one of promoting replication and adoption of programs to valuing 
adaptations and the role of exemplary programs in professional development. 

As with most misconceptions, these four contain elements of truth. In this case, those elements are 
based in the early years of the former system but are not accurate descriptions of the mature phase 
of the programs. As federal education programs go, the longevity of the National Diffusion 
Network and its validation mechanisms was unusual. Operations and practices changed a great deal 
over the twenty year period of the system's evolution. But relatively little recognition of those 
changes occurred in the community outside or, in fact, even for many inside the NDN. As with 
many programs, the Joint Dissemination Review Panel (the predecessor of the PEP) and NDN 
received the greatest amount of attention from researchers and policymakers during start up phases 
when plans were first put on paper and debate about how to proceed was active. 

Researchers still are more likely to find information that describes tlie first ten years of the Joint 
Dissemination Review Panel and NDN rather than the last ten years, overlooking the period that 
holds the most relevant lessons for the expert panel system, that is, experiences of validation and 
dissemination in the more contemporary school reform context. Unfortunately, much of what was 
written about the system in the past ten years was largely for in-house consumption. Drawing 
lessons from critiques of PEP and NDN operations without understanding how they operated in the 
last ten years is analogous to discussing changes needed in Title I without recognizing the ways in 
which schoolwide programs have changed Title I operations. 

The NDN's internal culture was closed and defensive as a result of the many political attacks on the 
program during the mid-Eighties, making it difficult for outsiders to obtain an accurate picture of 
operations. Newcomers and new ideas were not easily welcomed into the culture and articulating 
ideas for improvements in practice became profoundly threatening to some insiders. In the field, 
however, the "business" of validation and dissemination had actually changed substantially to keep 
up with the changing environment. 

Misconceptions about identification of programs 

For the most part, in the recent ten years, projects did not self-nominate for validation by PEP even 
with the appeal of a ready-made dissemination system in place and funds available for dissemination. 
While it was true that anyone could submit a program for validation, it was far from being an "open 
call" system. Actual practice was that many programs that applied to the PEP for validation had 
been carefully targeted as potential candidates, provided with training and guidance to present 
evaluation evidence in a credible fashion, and generally ntartured into the system. Part of RMC's 
role as the NDN technical assistance contractor was to identify potential candidates that matched the 
Secretary of Education's priorities, persuade them to apply, and provide the necessary assistance to 
assure successful validation. We found projects through literature searches, professional 
organizations and associations, intermediaries such as the education laboratories, and the state 
facilitator system. Technical assistance included help in reviewing evidence, creating evaluation 
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designs, identifying instruments, specifying appropriate analyses, and providing reviews of written 
submittals. 

The PEP/NDN faced two obstacles in identification of projects that will be faced by the expert panel 
system: (1) the incentives for validation (even during a time of specific funding for dissemination 
of validated projects) were not strong enough for some excellent projects to make the effort to apply; 
and, (2) very few programs, including well-funded large scale development efforts had collected 
evidence of outcomes in any systematic fashion. Finding incentives to encourage application for 
validation may be the biggest challenge ahead of the expert panel system. The recognition provided 
by promising and exemplary designations will have to carry substantial prestige in order to balance 
the work that will be required in preparing submittals; as we learned with PEP, money is not a 
substitute for prestige. Potential sources of prestige are: (1) the stature of the expert panelists who 
nominate and confirm, including both individual and institutional prestige; (2) the nature of the 
award and attendant publicity associated with the award; and (3) the access opened by the validation 
to influence within the field, the Department of Education, and OERI. 

PEP's insistence on evidence of meaningful outcomes — not on a particular evaluation design (see 
below) — meant that validation without substantial new work was out of the question for many 
potentially interested programs. The work required to collect evidence to build the evaluation case 
required two to three years for most projects. Once the pool of projects funded through Title III/IV 
programs in the Seventies (when evaluations had been required by program regulations ) was 
exhausted, it took several years to return to the point of having 30-35 viable candidates applying to 
PEP each year, viability being a combination of interest in dissemination and potential to document 
meaningful outcomes. 

The pilot expert panels that operated last year have already begim to experience the difficulties of 
finding projects with appropriate levels of evidence to qualify for validation review. An advantage 
of the proposed system is its ability over the long term to influence the larger fields represented by 
panels to encourage attention to evaluation during the design and funding of projects. In the short 
run, it is important to adjust the expectations of panel productivity, i.e., the number of candidates for 
validation, to the realities of the state of evaluation within particular fields. 

Misconceptions about evaluation requirements 

A major shift occurred in 1987 with the change from the JDRP to the PEP and the publication of 
new guidelines and criteria. Those steps encouraged PEP panelists and candidate projects to accept 
evidence of a wide variety of types in service of a broad array of outcomes, including changes in 
teacher and student attitudes and behaviors as well as institutional outcomes. The intent was to 
signal greater emphasis on the meaningfulness of outcomes and the robustness of evidence rather 
than on particular design approaches or types of evidence. 

Many prospective PEP candidates took advantage of the broader perspective. While the stereotype 
of the PEP as demanding only test score evidence continued to persist in some quarters, projects 
were successful in providing evidence not only in the form of more contemporary achievement 



assessments such as performance assessments, but also writing samples, content analyses of students' 
responses to inquiries, case studies, observations of teachers, parent and teacher ratings, social 
attitude scales, and records related to coinse selection and wages on entry into employment. As 
interventions became more complex, the variety of evidence increased to the extent that some PEP 
submittals were summaries of substudies. Of coinse, some candidate projects continued to use test 
data as evidence, often because achievement measmes were the most available form of relevant 
evidence, particularly when information was gathered retrospectively. And some PEP panelists 
continued to press for achievement tests as familiar measmes. 

The attempt to broaden notions of acceptable evidence faced two issues that the expert panel system 
v^ll also likely face; (1) reviewers may continue to express strong preferences for particular types 
of evaluation designs and evidence; (2) evaluation designs that are responsive to complex programs 
are challenging and expensive to develop and implement; for example, the conduct of multiple 
studies v^th a variety of soinces is much more costly than administration of off-the-shelf 
assessments. 

The first issue is important because the proposed expert panel system includes evaluation 
requirements that are very similar to the PEP's expectations but the new process suggests a 
diminished role for evaluation expertise on expert panels. In oin experience with PEP/NDN as well 
as with the Title I Recognition program, those least knowledgeable about evaluation sometimes hold 
narrow beliefs about what constitutes appropriate design and instrumentation, expecting for example, 
that the only type of acceptable comparison standard is a control group. Panelists without evaluation 
expertise may be the harshest critics of evidence gathered by innovative evaluation techniques, 
thereby limiting the success rate of projects that are "cutting edge." This caution supports the value 
of ongoing training for all panelists along with frequent communication around real examples. The 
idea of evaluation expertise available "on call" to expert panels is a good one; it will be critical to 
find ways that expertise is shared and becomes part of the experience of belonging to the standing 
and expert panels. In this manner, innovative evaluation practices can be disseminated along v^th 
promising and exemplary interventions. 

Inherent in the newly proposed system is the stepping up of programs from "promising" designation 
to "exemplary" status based on the strength of credible evidence. It is easy to imagine the advice that 
will be provided to "promising" candidates to craft designs that represent the most valid approaches, 
including, for example, the design of performance assessments tailored to project objectives. If 
candidates are similar to those we worked with in the past, they will need significant support to 
locate and retain evaluation assistance to prepare them for validation and then to take them from 
evidence appropriate for "promising" to the "exemplary" designation. At this point, the extent of 
support available for evaluation assistance is not clear; these demands could represent significant 
claims on resomces. It is important to clarify the role of expert panelists in providing evaluation 
support to candidates, i.e., how much is expected of them if they encourage nominations? Is it even 
appropriate for expert panelists to be providing evaluation advice? 
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Misconceptions about the value of projects 

There is the lingering sense in some quarters that the projects validated by PEP were not the 
iimovations really needed to improve American education. It is true that the PEP validated a very 
wide range of interventions — it did not limit review to projects of a certain scope or intended scale- 
up potential. Some of those validated projects were perhaps of little importance to wide-scale school 
reform. Certainly, however, many others have become nationally well-regarded as important 
educational interventions, e.g., Reading Recovery, CASAS, High Scope, Parents as Teachers, 
IMPACT II, Choices for the 2Ist Century, the National Faculty, City-As-School, Talents Unlimited, 
and the Program for School Improvement. Others never gained national reputations even though 
they might have been equally important catalysts for change at local sites. 

One issue about the perceived importance of projects that repeatedly arose during the search for 
candidate projects, particularly in the late Eighties, was the charge to find and validate effective 
"whole school" change models rather than the curriculum and professional development 
interventions for which the PEP/NDN had become known. At that point in time, the federal interest 
was in models of the processes that would lead to initiating and then sustaining significant structural 
and systemic reforms. Only a few comprehensive reform projects, including the Program for School 
Improvement (University of Georgia), the Outcomes Driven Developmental Model, and the More 
Effective Schools/Teaching Project, were validated and widely adopted. 

We made futile attempts to interest some of the better known comprehensive change models such 
as Levin's Accelerated Schools model or the Coalition for Essential Schools, in validation by PEP. 
In some cases, those models were expanding rapidly and they simply did not desire additional 
dissemination demands; in other cases, little formal evidence of outcomes was available. From 
some perspectives, the limited number of validated whole school change strategies marked a failmre 
of the PEP and NDN system as a school improvement strategy. 

Because the expert panel system, by virtue of its structure, seems to be focused more narrowly on 
curriculum and teaching projects and materials, it is bound to face similar criticisms. The highly 
specific nature of the pilot panels, i.e., gender equity, math and science, technology, early literacy, 
makes a bold statement that a wide variety of types of interventions can offer significant value to 
American education. Indeed, the requirements of the law that the expert panels review materials and 
research in addition to programs further suggests that even fairly focused interventions can produce 
valuable outcomes. How will a collection of these specific interventions be perceived in the larger 
school reform environment? Out of context, will an elementary grades math program and a high 
school physics unit and a method for combining phonics and whole language once again seem 
"trivial" as exemplary interventions? Or, do we now all feel more comfortable that specific practices 
can fit into an overall school improvement strategy? It is important that the expert panel system 
have an articulate position on how a wide range and scope of promising and exemplary interventions 
can function as "building blocks" for school reform. 

Misconceptions about philosophy of change 

Over the years, the basic PEP/NDN philosophy about how educators learn and how change 
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occurs evolved significantly — moving from the goal of faithful replication of highly specific 
curricular models to offering a source of professional development and expert consultation 
around specific interventions. 

The shared belief system during NDN's first ten years demanded faithful replication by adopters of 
the key elements that were characteristic of validated programs. True believers in this philosophy 
of improvement were reluctant to allow adopters to depart in any way from the original model; 
outcomes were guaranteed if replication was as close to identical as possible and program developers 
felt it was somehow cheating to condone alternative uses of validated programs. Training was 
required and emphasized "how to do it," purchase of materials was not permitted without training, 
and implementation was closely monitored. The strong belief in the value of personal contact with 
developers — ^the person-to-person quality of dissemination — ^was a further complement to the 
emphasis on faithful replication. 

But times change. In the late Eighties, programs seeking validation became more complex; more 
programs emerged from strongly developed research bases. Programs came to the PEP with an 
extensive dissemination history and with an array of dissemination styles and vehicles. For 
increasing numbers of programs, training and consultation began to be more about the principles 
behind strategies than about specific implementation steps. 

Unfortunately, the system as a whole never thoughtfully examined the evolution of the underlying 
philosophy of change that had occurred within the NDN community and, indeed, within the 
education community at large. Baggage from the days of faithful and unquestioned replication 
persisted in the language used to describe activities. The accountability structure remained tied to 
the notion of replication as did many aspects of validation procedures, i.e., expectations around 
revalidation, assumptions about the similarity of outcomes achieved across sites. Participants 
frequently expressed fmstration that their efforts in working closely with school districts to adapt 
validated programs to the context and developmental stage of local reform efforts went 
unrecognized. Everyone knew fundamental world views had changed but somehow there was no 
forum in which to fully acknowledge the full dimensions of the shift, particularly at the federal level. 
Perhaps, even more important, there was no way to articulate the transformation in philosophy to 
the outside education world which had come to be somewhat contemptuous of the simplicity of 
replication in the era of systemic reform. There was no easy way to explain that an entire system 
had changed its mind about the core principles that were once held so dear. This confusion 
ultimately undermined the credibility of the entire system. 

The implication for the expert panel system of this unfortunate history is the importance for 
designers of the new system to articulate clearly the philosophies of change and reform underlying 
the new system. System designers' central philosophies of how people leam and how schools change 
should inform procedures and operations of the expert panel system. 

The philosophy of change behind the expert panel system seems to be one of using models of best 
practice to inspire reforms. The emphasis on identification of worthy practices and strengthening 
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of evidence suggests beliefs that the quality of results is what is ultimately attractive to the consumer 
and that change is motivated by knowledge of the results that others in similar circumstances have 
achieved. This view is consistent with the contemporary notions of benchmarking as an 
improvement strategy. Indeed, several proposed aspects of the expert panel system support this 
philosophy, i.e., the emphasis on assessing worthiness by association with individuals with strong 
reputations in a field, the expectation of results jfrom multiple sites, and the expectation of 
information about contexts. The linkage to intermediaries, i.e., the education laboratories, ERIC, 
and the comprehensive centers, for the purpose of dissemination is consistent with the notion of 
providing a wide array of best practices as inspiration for reform. 

The change philosophy underlying the expert panels will be determined ultimately by the 
dissemination strategies that accompany and grow up around the expert panel system. And, it is 
really only in the use of validated projects by intermediaries and consumers that the philosophy of 
the expert panel system will become finally defined in the eyes of the education comm un ity. 

Summary: Enthusiasm for Features of the Expert Panel System 

The foregoing discussion provides a number of cautions about the operations of the expert panel 
system and it does not convey my personal enthusiasm for the many positive features embedded in 
the plans. 

Elements that seem to offer substantial improvements over past structures include: 

• the field-based structure of expert panels that encourages support of the overall 
system by a variety of fimding agencies, professional groups, divisions within the 
Department of Education; 

• the widespread involvement in recruitment of projects through the system of field- 
based experts; 

• the emphasis on content and quality of interventions within the review; and 

• the flexibility of procedures to tailor reviews to particular fields and types of 
interventions. 

Those and other features of the expert panels are excellent examples of learning fi'om past 
experiences and the expertise in other systems to craft an exciting experiment. 
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Background 

The 1994 reauthorization of the Office of Educational Research and Improvement (OERI) in the U.S. 
Department of Education mandated OERI to: (1) develop standards to designate promising and 
exemplary products, programs, and practices; (2) establish a system of expert panels to make 
recommendations on the above designations; and (3) coordinate dissemination activities and 
programs within the Department and other government agencies, as well as with associations, state 
education agencies, and local school districts. 

OERI has been charged with helping educators make more informed decisions about promising and 
exemplary products, programs, and practices as they attempt to implement these improvement 
efforts. The office has published a set of standards for public comment (June 1996) and is currently 
pilot testing two expert panels in the areas of gender equity and mathematics/science education. 
OERI has begun conducting research on existing review efforts that attempt to identify and share 
the best of what is available with consumers. 

This paper was commissioned to reflect on the system of expert panels to designate and disseminate 
promising and exemplary products, programs, and practices. It is related to research completed for 
a commissioned paper that examined review efforts in a wide variety of education organizations and 
topic areas (Luczak & Ruskus, 1997b) and a related “Lessons Learned” AERA paper (Luczak & 
Ruskus, 1997a). The goals of this paper are to provide reflections on what OERI has done so far with 
their system of expert panels, and to provide guidance on the dissemination of the programs they 
deem promising or exemplary. 

The paper will focus less on the specifics of the review process and more on future OERI 
dissemination efforts. Although the issues involved with the definitions of “promising” and 
“exemplary” will not be addressed, these distinctions may prove problematic in the future. Research 
findings from the commissioned paper authors (Luczak & Ruskus, 1997b; Borich, 1997) show that 
the some in the “field” are unfamiliar with these terms. This paper attempts to answer certain 
questions surrounding incentives to apply for promising or exemplary status, two potential vehicles 
for dissemination, repercussions from widespread use of such a dissemination system, financial 
limitations of the overall effort, and the recently released draft statement of work for a dissemination 
contractor to help OERI with these efforts. 
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Where’s the Carrot? 

Perhaps the biggest and most difficult problem to solve is how to get enough programs to submit 
applications to the expert panels (see Campbell, 1996; Maxwell, 1997). Programs do not apply 
because they either (1) do not know about the process, or (2) they do not have a good reason to do 
so. In the first case, it is safe to assume that the large majority of programs or projects across the 
country are not aware of current OERI efforts. This problem can only be remedied through an 
effective public relations campaign and could cost a great deal of money. OERI must take advantage 
of “word of mouth” efforts and leverage all involved with these efforts to encourage programs to 
apply to the panels. 

The second case scenario exists because programs rarely find an enticing answer to the question, 
“What’s in it for us?” Frequently, any potential positive answer to this question goes unheard 
because time and/or resources are too short for anyone to spend time filling out the necessary 
application. Potentially effective incentives for program application besides money include the 
following: (1) recognition through newsletters or the dissemination of accepted program profiles, 
often via the Web; (2) networking opportunities that occur at conferences or award ceremonies; and 
(3) professional development/technical assistance experiences from “expert” site visits and the 
application process itself (Luczak & Ruskus, 1997a). 

The next most pressing problem facing many programs is their lack of achievement data (required 
for exemplary status), and the difficulty or high cost of obtaining such data. Many of the review 
efforts studied emphasized to potential programs the professional development experience that 
comes from completing an application (Luczak 8c Ruskus, 1997b). In fact, a few of the efforts 
required programs to complete self-assessments as part of the application process, and this was an 
incentive for programs in many of the accreditation types of reviews. The expert panels should 
emphasize the professional development aspects of undergoing a review more in their marketing 
materials. 

OERI should also promise to provide technical assistance to all programs that apply for promising 
or exemplary status. Programs judged to be promising could be given a small grant to help them 
collect achievement/evaluation data or do whatever it takes to resubmit and obtain exemplary status. 
A thorough report of the expert panel’s review arid a plan for improvement should be provided to 
the applicant if possible (with references to an evaluation-fnendly Web site, see below for more 
details). The program’s incorporation of “empowerment evaluation” techniques — ^basically an 
internalization of evaluation principles and processes (see Fetterman et al, 1995 for further 
explanation) — should also be advocated. As Carter and Schilder (1997) write about foundations, 
“participatory and empowerment evaluations not only helped identify promising models but also 
generated information that was judged useful to the programs.” Regardless of whether or not the 
program resubmits, the review process will definitely help the program improve, and the potential 
money for self-evaluation activity will help attract more applicants. 

A “Consumer Reports” for Promising and Exemplary Programs? 

One dissemination effort with a potentially high payoff is a “Consumer Reports” type of periodical. 
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These could be established by subject area. For instance, the first issues would include products, 
programs, and practices related to gender equity, mathematics/science, early reading, and education 
technology. “Consumer Reports” issues could also be produced for special education, vocational 
education, early childhood education, postsecondary education, educational equity, gifted and 
talented, etc. 

A consistent format across expert panels should be agreed to by OERI, and each expert panel should 
be charged with the development of a mini-magazine or newspaper type of publication in their 
respective subject areas. A good version of a “Consumer Reports” issue in a certain subject would 
include the ratings of all projects reviewed, or at least those designated promising or exemplary. 
Criteria for program comparisons will be individual to each expert panel, but a comparative matrix 
display of the most relevant criteria would be very informative. A consumer interested in choosing 
from among the exemplary programs could then do a direct comparison along certain criteria, and 
determine the program most appropriate for his or her situation. Depending on the specific subject 
area and the type of submittals, the issues would be oriented at either one of two levels, or both: 
program information primarily of use for state or federal level policy-makers, and products and 
practices primarily of use for district or school level administrators or teachers. 

More generic articles on the entire system of expert panels and an update on OERI’s dissemination 
efforts would be effective as an introduction to each issue. Also critical is an explanation of the 
review guidelines used. And perhaps most importantly, example reviews of a promising and an 
exemplary program should be laid out in detail, because otherwise the review criteria remain a list 
of regulations, disconnected from many of the readers. Potential program applicants can get a better 
sense of what it means to fill out the application and go through the review process at the same time 
they come to a better understanding of what the expert panel is looking for in a promising or 
exemplary program. A reflective article by an expert panel member regarding the review process 
would be both interesting and informative. Issues depending on their subject area could contain 
more in-depth reviews of certain individual programs, a focus on what works in elementary schools, 
or urban schools, or special issues focusing on one narrower strategy. 

The marketing and graphic design involved with these “Consumer Reports” issues will be critical. 
Unfortunately, what many people look at first when picking something up is very superficial — ^the 
color or design on the cover, the page layout, the length, etc. Although these issues do not have to 
be long, they must look nice, and if possible should be printed in color. To help defray costs, 
subscriptions to these issues could be purchased for a reasonable price. It would be relatively easy 
to market these issues at the school, district, state, and federal level. Of course, it might make sense 
to give them away free initially to build up a readership. (Advertising opportunities are discussed 
later in the paper.) Each expert panel could produce a specific content issue once a year, depending 
on how many programs submitted applications and how often the applications are reviewed by the 
panel. 

Who Would Use a Web Site? 

Slavin (1996) writes that teachers desperately want better methods and materials, because they know 



they need to use more effective instructional methods in order to improve student achievement. 
Some teachers have already begun to turn to the Internet for help in their classrooms. A Web site 
dissemination effort would be different from the “Consumer Reports” issues described above 
because it would provide access to promising and exemplary products and practices, and be more 
practitioner oriented. The Web site could include subject area specific, structured product and 
practice profiles that provide a brief description of the activity or program, its age appropriateness, 
other contextual factors, etc. 

The most exciting feature of this Web site would be the interactive feedback it encourages. Anyone 
who attempted to implement a promising or exemplary program of any sort would be strongly 
encouraged to report their experience (and maybe even paid a nominal fee). An intake process 
would have to be designed to screen (e.g., the contractor described below) and post all feedback in 
an orderly fashion. For example, a teacher using an “exemplary” mathematics/science lesson plan 
may report what worked best, what could be improved, and what he or she would change the next 
time. Extremely negative comments would be noted internally but not posted. If there was a lot of 
negative feedback surrounding a program, its status could be reviewed. 

Another useful feature of such a Web site would be demonstrations of successful evaluation plans, 
instruments, and reports. SRI is currently creating such a Web page for the National Science 
Foundation’s Division of Undergraduate Education programs. That way, new projects can utilize 
this evaluation resource library when they are beginning to plan their evaluation strategies. 
Annotated examples of plans, instruments, and reports in their full text versions will be included in 
these pages, along with more generic evaluation information. A generic description of the expert 
panel review criteria, the review process, and then review examples of both a promising and an 
exemplary project would also be worth including (as in the “Consumer Reports”). 

The Web site should be linked to relevant Web sites of ED regional labs, ERIC, technical assistance 
centers, and others that would benefit from these promising and exemplary product profiles (Klein, 
1997). The other Web sites could include some or all of the profiles on their sites, advertise the 
expert panel URL address, or perhaps share some of their “best practices” with the expert panel Web 
site. The system of expert panels and the Web site should also try to establish ties with existing 
teacher organizations like NCTM, NSTA, NEA, AFT, and ASCD. Representatives from OERI 
and/or the expert panels should attend their national meetings as an effective way to get word out 
to teachers about the expert panels and designated promising and exemplary products and practices. 

The state of Montana currently uses a distance telecommunications network called Metnet. It is 
maintained by the state, with much help from MCTM and MSTA, and allows teachers to use search 
engines to find certain lessons. It has been quite popular with early career teachers, and it may be 
worth examining Metnet’s successes and failures before implementing a Web site. Both the 
“Consumer Reports” issues and the Web site need to be test marketed with potential practitioners 
and policymakers who know which dissemination materials they will and will not be able to utilize. 
Getting feedback from real consumers (especially asking them what they would find most helpful) 
before either of these efforts are implemented is critical. 
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Repercussions from Wide Use of this Dissemination System? 

Widespread use of the dissemination options described above could have overarching implications 
for our current education system. The 21st century could usher in teachers who take to the Internet 
to look for potential promising and exemplary lessons (products or practices) in certain subject areas, 
state policymakers who turn to the most recent “Consumer Reports” in vocational education in order 
to make an informed choice about what type of school-to-work program to implement statewide, or 
even Superintendents who look at the education technology issue of such a magazine to help inform 
a district’s technology plan. 

A widely successful dissemination system for promising and exemplary products, programs, and 
practices could have economic implications for many players in the education system. Curriculum 
material and textbook publishers would pay close attention to such a ratings system (especially if 
a “Consumer Reports” on such materials was widely disseminated), because it could strongly 
influence their market share and profitability. Such ratings could also influence the funding 
decisions made by foundations, or lead to a redirection of federal fimding (e.g.. Title I, Eisenhower) 
toward exemplary programs. Success could also lead ED to their legal counsel. Hopefully, ED 
could collect advertising revenue from any and all “Consumer Reports” issues they produce, and use 
this money for the expansion of their expert panel system or for program application incentives. 

As the rankings of “promising” and “exemplary” become better known nationally, programs would 
have more incentive to apply and try to earn the top distinction. Extensive popularity of this 
dissemination system would result in better awareness of program evaluation and self-evaluation 
methods. An extensive technical assistance package might be sent to programs who are thinking 
about applying to the expert panels. A few of the most recent “Consumer Reports” in the related 
subject area, review examples of a promising and an exemplary program, and a sample evaluation 
plan and report (and the Web address) could be included in such a package. 

What are the Financial Limitations of this Effort? 

It appears that obtaining major federal funding for these efforts will remain a challenge. Once a few 
of the expert panels are running smoothly and have designated a first round of promising and 
exemplary programs, OERI should approach foundations or other government agencies for a large 
infusion of capital if their federal funding level remains low. The dissemination efforts described 
above would benefit greatly from foundation funding (or multiple fimding from several federal 
agencies) for the use of real-world marketers, graphic artists, etc. 

Such a proposal would be enticing to foundations for the following reasons: (1) OERI will have 
done the extensive legwork needed to get the project off the ground; (2) a dissemination contractor 
will be in place (see next section) to help with both administrative and analytic tasks; (3) the system 
will be popular with both content and knowledge utilization experts (primarily located in 
Washington) and largely unknown to the larger education community; and (4) funding professional 
dissemination-type activities could greatly increase the number of professionals learning from these 
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promising and exemplary programs and the number of program submittals, as well as the public’s 
overall awareness of such a system. 

The system also needs to avoid the perception that everything is going on in Washington. Regional 
meetings of expert panels would be helpful, or even basing a few of the expert panels out of different 
geographic regions would be a start (similar to ED’s regional labs, research centers, and ERIC 
clearinghouses). Obviously, better funding is necessary to convene regular meetings of the expert 
panels, especially at different geographic locations. Diverse homes for the expert panels would also 
help spread the word about OERI’s efforts and allow for better marketing of different regional 
programs. 

What Can a Contractor Do to Help? 

OERI is requesting proposals for “the design and implementation of a technical assistance activity 
which will provide logistical, administrative, and analytical support for the Department’s expert 
panel system for designating promising or exemplary educational programs and sharing this 
information with the broader educational commtxnity and the general public by the most effective 
means possible.” (OERI, 1997) The recently released draft statement of work for the “Dissemination 
and Evaluation for Promising and Exemplary Programs” included fom major goals: (1) provide 
logistical support and expert advice for creation of two new expert panels (approximately 15 
members each), one in education technology and one in early reading; (2) provide support and advice 
for the standing panel, including training for the members of the standing and expert panels; (3) 
provide support and advice for the continuation of the mathematics/science panel; and (4) implement 
a comprehensive system to disseminate promising and exemplary educational programs and a 
summative evaluation of dissemination efforts. 

The technical assistance/dissem in ation contractor will work with OERI for five years, which is 
critical because it will allow them to conduct long-range planning. The statement of work indicates 
that the contractor will recruit, train, and support all panelists and reviewers (some 260 total 
reviewers). They will hold many 2- to 3 -day meetings and trainings each year, all in Washington. 
Additional expert panels will be added for other content areas as funds become available. The 
contractor will also be responsible for helping solicit promising and exemplary programs for review 
by the panels, and for maintaining a “state-of-the-art” homepage on the Web. It appears from the 
statement of work that the contractor will be relatively well-funded, unlike the support contract for 
the pilot expert panels this past year. The selection of this contractor and the quality of their long- 
term dissemination plans will be most critical to the success and longevity of the expert panels. 

Conclusions 

The key to the survival and prominence of the expert panel system is in the details. Advertising and 
marketing savvy will be critical to the success of the expert panels which will designate promising 
and exemplary programs, especially in drawing consumers and applicants to the system. The initial 
growth of the panels and the number of programs identified as either promising or exemplary will 
play an important role in determining the popularity of the expert panel system. 
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Once initial numbers grow, dissemination strategies will become critically important. The 
“Consumer Reports” issues and the development of an effective Web site are critical steps to take 
after test marketing them with real consumers. An influx of funding (either gover nm ent or 
foundation) is necessary for the system of expert panels to have long-standing, large-scale impact 
in the education community. Hopefully, a busy and well-off contractor will be able to get things 
started. OERI should think, however, about hosting panel and training meetings at different 
locations across the country. While in these diverse locations, they could conduct outreach and 
public relations activities to help spread the word about the system of expert panels. 
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Dissemination and Evaluation Issues Related to the System of Expert Panels 

Dr. Gary Borich 

Univ. of Texas, Austin, March 31, 1997. 

Q. How can practitioners help the Department understand the strengths and weaknesses of the 
programs they are using? 

Q. How do we involve consumers in the total review process as well as in making sure that they 
receive comparative evaluation information that will help them select what is best for them? 

A. One way of encouraging consumer participation in the review process is to require consumer 

reactions as evidence of effectiveness for promising and exemplary status. A distinction is 
made, however, between participant responses to a program in an experimental context 
and consumer reactions in a naturalistic setting. Since most evaluations tend to be 
comparative— or summative, consumer reactions in a naturalistic setting can serve the 
formative purpose by identifying areas of desired revision and modification to a program 
or product. Therefore, consumer reactions to the program or product, as it is being 
implemented in the field, should be among the data required for achieving promising or 
exemplary status. 

These data should be targeted to identifying areas of perceived strengths and weaknesses of 
a program— as opposed to its overall effectiveness— and go beyond statistical indices that may 
mask variation in responses to a program-by specific subgroups of consumers. Qualitative 
indices of consumer satisfaction, case studies, interviews and vignettes of participant 
dialogue that capture consumer confidence would serve this formative purpose and provide 
for the consumer natural language benchmarks for selecting what is best. 

Q. Are there different ways to disseminate exemplary versus promising programs and if so what 

are they? 

A. No. It is important to maintain a distinction between exemplary and promising based on the 
degree to which quantifiable performance standards for a program or product are met. 
Different dissemination strategies would obfuscate the distinction by implying that a 
promising program or product is qualitatively different than an exemplary one, when, in fact, 
the distinction is one of degree. In that manner the Department of Education conununicates 
that all promising programs and products can and should be expected to move up to 
exemplary status by achieving the same standards, but quantitatively exceeding the level at 
which the standards were met for promising status. Different dissemination standards would 
be contrary to maintaining a quantifiable continuum between promising and exemplary that 
is demarcated by the same intervals of improvement. 

Q. What should the U.S. Department of Education do to encourage organizations of every kind 



to participate in its dissemination system? 



Q. Given the two-tiered designation of promising and exemplary, what kinds of incentives 
should the US. Department of Education encourage and/or provide to entice people to apply 
for the designation of promising for their programs? 

Q. What kinds of incentives should there be to help people who have a promising program more 
toward the designation of exemplary? 

A. The Department of Education should set aside funds for programs and products designated 
as promising for the purpose of evaluation, in the case of promising, and for further 
development and dissemination, in the case of exemplary. The incentive for participation 
would be that, if chosen as promising, an agency could compete for funds for evaluations that 
would be specifically conducted to show that the program or product has achieved standards 
at the exemplary level. Although funding would be competitive among programs achieving 
promising status, agencies would be encouraged to conduct their own evaluations when not 
funded by the Department of Education, guided by their proposal submissions. In addition, 
fi"om the evaluations that are funded other projects would see how to collect data representing 
the standards and the degree of data required to move fi"om promising to exemplary and to 
design evaluations accordingly. Likewise, funds could be designated for programs and 
products designated as exemplary for the purpose of further development so as to be 
generalizable to (a) other high priority target groups, for example, at risk, and/or (b) other 
topics, levels of schooling or content areas. 

Q. What kinds of incentives should there be to help people who have a promising program move 

toward the designation of exemplary? 

A. One incentive for publishers and developers would be for the Department of Education to 
formally communicate the promising or exemplary status of programs and products to 
potential adopters. The manner in which promising and exemplary programs and products 
are communicated to the public could include the following: 

One form of communication and dissemination would be a publication specifically designed 
to identify and describe promising and exemplary programs and products that would be made 
available to the public. Potential users could subscribe to a nationwide mailing list to receive 
the publication. 

A second form of communication would be to notify the professional associations to which 
the program or product may have relevance, indicating its promising or exemplary status. 
Associations fi^quently provide awards, ceremonies and recognitions to projects in their field 
that have attained distinction. Announcements at professional meetings and in newsletters 
are customary channels of communication to association members. 
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Third, the Department of Education could service and periodically update a Web page on the 
Internet indicating the programs and products achieving promising and exemplary status. 
Entries could be organized and cross- referenced by topic areas, audiences and levels (for 
example, populations, grades, levels of schooling, institutions, etc.) to individualize access 
for the user. With each entry, a description and the source of purchase could be provided to 
create the "sales" or user incentive for projects to apply for promising or exemplary status. 

How important are external incentives to the system? 

Very important. The system should provide funds for the further evaluation of promising 
programs and for the further development of exemplary programs. Programs and products 
designated as promising should be able to compete with other promising projects for funds 
specifically targeted to the further collection of data according to the four standards. 
Evaluation of the proposals would be based on the likelihood that the data collection 
proposed would provide unambiguous evidence whether or not the program or product 
achieved exemplary status. In the case of exemplary programs or products, funds should be 
competitively provided for expanding the program or product to a new and needful target 
population, level of schooling, or content area, as determined by Department of Education 
priorities. The criteria for judging competitive proposals for programs and products of 
exemplary status would include the extent to which the need for the new population, level 
or content has been documented and the likelihood that the theory or rationale guiding 
development of the program or product would be generalizable to this new context. 

What are the best ways to differentiate between "promising" and "exemplary"? 

The best way to differentiate between promising and exemplary is to quantitatively define 
the difference between these designations with regard to evidence of effectiveness/success, 
quality, significance, and usefulness. It is important that these same standards apply to both 
promising and exemplary, but that the degree of validity, reliability and generalizability of 
the evidence quantitatively differ across designations. This will entail that the Department 
of Education construct "data sets" in the form of profiles of evaluation data for each of the 
four standards applied to example programs and products to illustrate the gradation from 
promising to exemplary. In other words, a two tiered continuum of improvement would have 
to be achieved with exemplars of data that would be relevant to the programs, products, 
practices that are of concern to a wide variety of agencies. As programs and products 
actually achieve promising and exemplary status, their data sets across the four criteria of 
effectiveness/success, quality, significance, and usefulness could replace earlier, more 
illustrative data sets. 

To what extent will the same review criteria and procedures work for identifying practices 
and policies as well as programs and products? 



A. 



The criteria must be flexible. Although the same four standards must apply to practices and 
policies as well as to programs and products, reviewers will find that the nature of the 
evidence will be different across at least some of standards for practices and policies versus 
programs and products. For example, controlled experiments and treatment manipulations 
are less feasible when examining practices and policies as when evaluating programs and 
products. The former are often systemic to the institutional structure and not easily 
compartmentalized for study as are programs and products. Also, practices and policies due 
to their seamless integration with the system of which they are a part often can not be 
"withheld" from one group and applied to another. Therefore, expectations for the "data sets" 
from practices and policies must necessarily be different from those for programs and 
products. This is why example data sets comprising profiles of evaluation data from the field 
will be important to ground the standards with the constraints of the workplace in which the 
practices and policies occur. 

Q. How do we develop a comprehensive system of expert panels which has an impact on federal 

and perhaps even foundation support decisions for the continued evaluation and revision of 
what is designated as promising and the active dissemination of those designated as 
exemplary? 

A. One way to encourage continuous evaluation and revision of programs and products 
submitted to the FindBest System would be to allow programs and products not selected as 
promising or exemplary to remain in a queue for further consideration pending an updating 
of evidence pertaining to the four standards. In this maimer participants may be more 
inclined to update their evidence, acquire new evidence and to increasingly tailor it to the 
four standards during subsequent cycles of the program or product. Programs and products 
that have updated their evidence would receive priority, i.e. less lag time, in being reviewed 
a second time over projects that were being submitted for the first time. This could provide 
the incentive to collect more data targeted toward the standards and would increasingly 
familiarize the participant with the criteria and standards used for deciding promising and 
exemplary status. 

A second consideration is to have the Department of Education commission "guidelines for 
the evaluation of promising and exemplary programs, products and practices." Many projects 
may not know or feel comfortable with the evaluation procedures that yield the quality of 
data required for determining promising and exemplary status. Evaluation guidelines would 
give direction to their data collection efforts for obtaining appropriate "data sets" targeted to 
the standards. The evaluation guidelines would be valuable to projects in gauging the level 
of commitment they must have to succeed as promising or exemplary and would, necessarily, 
persuade less serious contributors from overloading the expert panel system . 

How can we provide incentives for publishers, developers and users to collect and share 
information on what works or doesn't work for whom so that all decisions can be based to 
some extent on evidence of effectiveness, not Just on popularity or sales? 
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The assumption is that "sales" numbers do not reveal how the program or product performs 
with regard to the four standards. Yet, sales and popularity are relevant to the standards, since 
information of a product or program's effectiveness/success, quality, sigiuficance and 
usefulness are related to these figures. By encouraging the involvement of publishers and 
developers in the FindBest system and, thereby, enhancing their familiarity with the 
standards“they may become sensitized to the data required by the standards and provided 
by their constituent users as part of field trials, naturalistic observations and consumer 
satisfaction studies. 

In other words popularity or "sales" is a legitimate criterion in determiiung promising and 
exemplary status. The problem is not with popularity or sales as an index of effectiveness, 
but with encouraging publishers and developers to go beneath the "numbers" and to collect 
evidence as to why the numbers are justified as an index of effectiveness vis-a-vis the four 
standards. Publishers and developers routinely have access to these indices of effectiveness, 
for example fi-om pilot-users of their programs or products, but often fail to acquire these data 
fi-om constituent groups in favor of the easier to achieve bottom-line sales numbers. By 
including publishers and developers in the expert panels, they will see the importance of 
these data for determiiung promising or exemplary status and the sales implications of the 
data behind the broader indices of effectiveness promoted by the four standards. 

How can associations and other constituency groups assume ownership for much of the 
review and dissemination processes? 

Associations and constituency groups should be contributors to the review process, and be 
represented in the composition of the review panels. Their most obvious presence would 
include participation in the review panels, but need not be limited to that role. Associations 
and constituent groups should be involved in updating the standards as experience warrants 
and in providing profiles of evaluation data across the four standards at the promising and 
exemplary levels. These groups are among the most able to provide example evaluation data 
sets representing realistic portrayals of the performance of a program or product in the field 
that encourage identification with and ownership of the review and dissemination process 
by specific audiences. Their involvement should be based on the sharing of data sets and 
degrees of evidence for the four standards with respect to programs and products of their 
specific constituency. The profiles of evaluation data, i.e. "data sets," they contribute across 
the four standards should be a part of the "guidelines for evaluating exemplary and promising 
status" and in discerning the distinctions among promising and exemplary.. 

What are new ways that the US. Department of Education could spotlight exemplary and 
promising practices so that teachers, principals, and superintendents can have informed 
access to those programs and use them effectively? 

Given the two-tiered designation of promising and exemplary, what kinds of incentives 
should the US. Department of Education encourage and/or provide to entice people to apply 



for the designation of promising for their programs? 

What kinds of incentives should there be to encourage people to apply for either category? 

How do we involve consumers in the total review process as well as in making sure that they 
receive comparative evaluation information that will help them select what is best for them? 

What could be done hnmediately and what would need substantial planning and resources? 

How could these organizations best be involved? 

What kinds of organizations should the Department target as potentially the best cooperators 
in this effort? 

How can the federal government provide leadership in its support of expert panels in 
different areas where the resources and needs are different? For example, how can it build 
on ongoing review activities such as those in mathematics and science education and develop 
ongoing review activities in other areas such as gender equity? Should the various expert 
panels be supported equitably? 



The Expert Panels to be Charged with Identifying Promising and Exemplary 
Educational Programs: Issues and Suggestions 

Janet Carter, Ph.D. 

The Office of Educational Research and Improvement (OERI), U.S. Department of Education 
(ED) is sponsoring a system of Expert Panels (EP) to designate promising and exemplary 
programs. The purpose is to aid educators, parents and the general public in their search for 
educational improvement . As the panels have been designed and their procedures refined, 
much research, review and exchange has taken place. An amplifying paper has been requested 
from some of those who have been privy to these discussions. 

The following comments constitute my suggestions and are drawn primarily from experience in 
the philanthropic sector. They are arranged around perceived needs of the panels as follows: 

I. Gaining submisions of the panels 

II. Gaining commitment from experts to join and support the work of the panels 

III. Conducting the operations of the panels 

IV. Disseminating the findings to the panels 
I. Gaining submissions of the panels 

Submitting one’s work for assessment is a pursuit that has costs in emotional anxiety, staff time 
and production. To minimize or contain these costs would appear to be a major task of the 
panels organizers; the following approaches might be considered: 

• Enlist advocacy groups speaking for teachers or public authorities such as school 
districts or SEDs to provide nominees for EP consideration. Some of these 
organizations would already have lists of “best practices” or “teachers of the year” 
and consequently the submission of names would be a relatively low cost effort 
for them. They would not be asked to reformat or oversee the preparation of the 
submissions. Their role would be largely honorific and stress-free. Such 
organizations, for example, would not be asked to choose from among their 
members or constituents since decisions about the designation of excellence 
would rest with the Expert Panels. 

• Newsletters and meetings of such organizations or bodies would be invited to 
examine the plans and procedures of the panels, preferably through an interview 
format. The purpose would be to gain insight into the perception of the 
Panels’work from the field, as well as to encourage submissions. 
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• Membership on the panels might include organizational designees but this would be 
distinct from what is being described here. That is, the same organization would not 
carry out both functions; nominating and being represented among those who are 
judging entries. 

• Outlets, and possibly nominations, would be sought from other than the established 
educational community. Youth work and parental involvement groups share the EP 
commitment and could be approached. An example of such an outlet which might 
be utilized is Youth Today. The Newspaper on Youth Work' 

• Enrolling classroom teachers and school administrators as potential submitters is 
clearly a central need. This may need to be accomplished through outlining concrete 
benefits to be gained through designation by the EP. 

• The discussion at AERA in Feb., 1997 stressed that practitioners feel the need for 
help in preparing their entries and, most particularly, in meeting the evaluation 
requirement. Possible funding of program or evaluations of submittals would be a 
powerful inducement but even if that was not immediately possible, there would 
be value in such designation.. One approach would be to assign each entrant a 
buddy - a staff member of ED, a volunteer from a professional organization or a 
representative from a previously panel designated organization to encourage 
completion of the application process. While it would be unrealistic to ask such 
Individuals to conduct extended coaching, they could alert their practitioner(s) to 
evaluation materials, how to manuals, examples of past entries to the EP process 
and be available for supportive electronic check ins. The practitioner would commit 
to initiating such calls or e.mails at agreed upon dates and times. The goal would be 
to assist them in overcoming natural resistance to completing application materials, 
including gaining the involvement of others. 

• Another aid to practitioners would be the creation of local working groups of peers. 
A graduate teachers’ education program or community facility might be willing to 
offer space and encouragement for this activity A foundation associated with 
educational reform might also be the venue for time -limited preparation of entries. 

• For the submitters, the professional standing of EP members will be important. An 
inducement to enter would be provided by biographical outlines of the current 
board. Excerpts of their writing, interviews with them - including why they believe 
in the work of the panels - supplied electronically would be a low cost way of 
connecting people in the field with those who will be making the decisions. Higher 



1 Available from Youth Today, 1200 17th St. NW 4th Floor, Washington DC 20036-3006 
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cost encouragement could be supplied by giving full copies of a book or other 
writing, a video interview with a panel member or with the whole group. The goal 
would be to provide professional development linked to the preparation of an entry 
and to hmnanize and acquaint often isolated professionals with engaging and 
significant people in their field. 

• If financial support is to be forthcoming or can be reasonably anticipated, this 
should be outlined clearly. Impact II, Teachers Network has found that even 
quite small grants to teachers are important incentives and often creatively used. 

• The format of the application and the process of completing it may be turned to the 
advantage and enrichment of entrants. For example, if the application had to be 
completed by several different perspective holders, e.g., a school principal, a public 
health official, a student, a key complaint of many program people - the diverse 
audience and ineffectively linked decision makers with whom they deal could be 
addressed. The act of completing the application would expose key participants to 
each others’ thinking in the context of lauding the program. 

• A site visit following designation of the finalists, if that form is used, may be 
daunting but is also rewarding if site visitors are well prepared and supportive. 
Individuals from formerly designated programs would be good contributors. 

• The questions asked on the application should contain some that stretch into the 
future, thereby intriguing the respondent as well as reaffirming the core valuesof 
the program. E.g., “Where would you like to see this program in five years: what 
would be most important to you in terms of its accomplishments?” Is a practical but 
also optimistic question and acts to engage the writer. 

II. Gaining commitment from experts to join the panels 

The desire to improve the field and to participate with others they respect would be the 
strongest motivations for panel members to take part . The task of making selections is 
taxing , attention will need to be given to utilizing panel members time well and recognizing 
their contributions. Several aspects of the intellectual or professional rewards of 
participation may be suggested. 



• The Panels if composed of able people from overlapping but different 
disciplines or vocations would provide stimulus dming group sessions. 

• To expand the incentives, panel members could be encouraged to bring 
professional concerns of their own into the meetings shaped by the needs of 
reviewing the strengths and weaknesses of the applications. An example of 
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this kind of professional question is evaluation. What is to be encouraged 
and what do we know about the results to be achieved? Many program people 
and professional evaluators would now contend that participatory evaluations 
are necessary for program improvement and that much from semi 
experimental or experimental designed studies has not been used. Weiss and 
Greene argued as early as 1992 that “it is time for empowerment-oriented 
family support and education programs and their funders to move beyond 
experimental evaluation as the only legitimate, if often admittedly too 

expensive or otherwise undoable approach it is time to take some risks 

in order to make evaluation a part of the empowerment process, not a 
disconnected activity done by and for outsiders”.' At the same time as 
interest increases about the promise of such evaluations and their value as 
a source of technical assistance and professional growth for the staff of the 
agencies involved, questions remain about the results it can produce. Some 
research in this area is being conducted by Innonet funded by the Robert 
Wood Johnson Foundation. ^ This is a substantive professional question that 
could be addressed in the context of the Expert Panels contributing to the 
value of being a member of one. Among the foundations exploring 
empowerment evaluation are the Hogg Foundation for Mental Health in 
Austin, Texas and the Edna McConnell Clark Foundation in New York City. 
It would be useful to engage this wing of the funding community in 
discussion of the contribution of participatory evaluation. 

• This last suggestion leads naturally to an earlier mention of including 
organizational members on the EP. If funders are to participate they will 
need to do so without committing their Boards to particular subsequent 
actions. Membership selected by a professional association such as the 
Grantmakers Evaluation Network might signal a strong interest in the 
operations of the EP without suggesting the decisions of particular 
philanthropies. 

III. Conducting the operations of the paneb. 

The panels will work best if members believe their time is valued and that they grasp clearly 

the procedures being followed. There is no blinking the fact that what is called for is 



' Heather B. Weiss and Jennifer C. Greene, “An Empowerment Partnership for Family Support 
and Education Programs and Evaluations”, Family Science Review. Vol. 5, NO. 1 & 2, Feb. and 
May, 1992, p. 131-149 



^ Innovation Network, Inc., 1001 Connecticut Ave., NW, Wash DC 20036, (202) 728-0727 
FAX (202) 728-0136 



outstanding staff work. This need can be examined under several headings: 



Consider relieving the voluntary Chairperson from the job of conducting the 
work session on the selection of designated projects . The staff member who 
has processed the entries would establish the time line and enforce the agreed 
upon procedures. He or she would not take part in the consensus on selection 
or vote , if that form is being used. This may appear a radical idea but is 
drawn from the successful conduct of award juries and offers substantial 
advantages. See Appendix Two for outline of hypothetical work day. 

Staff should administer a time line and allocation of tasks which has been 
agreed upon by the Panel and should have the authority to remove or reassign 
jobs which panel members have not concluded. A great boon to this 
endeavor is the fact that panel members will be devoted to it. By definition, 
they will be people who very much want to uncover and hold up outstanding 
projects. They may professionally have been challenged to cite “what 
works” and felt frustrated in their attempts to respond to powerful critics of 
public education. Their enthusiasm for the EP effort may however expose 
them to the hidden rock of volunteer service: unacknowledged overextension. 
They may offer to read or do more than they can sustain. The result, under 
traditional procedures, is that basic communication or administrative tasks 
don’t get done and - once initial enthusiasm is consumed - a sense of 
disconnect or irritation sets in leading to withdrawal either by themselves or 
panel colleagues. 

The reverse side of the sternness of, is that sessions of the EP should be 
pleasurable. Everyone can remember when they worked very hard and had 
fun and other times when they definitely did not. The differences between 
the two experiences can be fairly plebeian. They include food and 
companionship. I once interviewed the survivors of an innovative advocacy 
effort. They had taken on a daunting task, labored for years and now were 
looking back from the security of being established. What had made them 
stick with it? “Well, said one veteran, “the sandwiches were very good”. 
Margaret Mead put it another way when she was training development 
workers, “There should be enough money for a shared cup of tea”, she said. 
Eating takes place in a space; could the meetings be held in vmexpected, 
meaningful places? One learns something from being in a colleague’s home 
and enjoying the books on the shelves and one is reminded of what one cares 
about by being in a downtown school or a nonprofit youth center. Another 
factor to be considered is the ambiance early established at the meetings. 
Fortunately, there are now many people who are skilled at getting meetings 
started with a period of mutual recognition and shared approval. The Panel 
would benefit from this practice. Despite the fact that their time together will 
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be extremely limited, attention to the full inclusion of all participants will 
serve as important longevity insurance for the enterprise. 

• The tone and effectiveness of the operations of the panels will be all the more 
important if, as suggested earlier, panel membership is selected in a minority 
of cases by organizations, that is, if professional associations with an interest 
in finding best practices in education are invited to select members. This 
suggestion introduces some immediate questions: what if “institutional “ 
members from the Education Commission of the States or the American 
Education Research Association or Grantmakers Evaluation Network turn out 
to be unreliable, strikingly different in orientation or disruptive to the 
collegiality of members with long standing ties? Two answers may be 
offered: rotation and recognition of the need for allies. A limit of no more 
than two years service for an organizational member could be set. (Whether 
or not this requirement should be enforced on all members is not addressed 
here). No matter how imperfect the fitm, therefore the situation would not 
go on too long. But more positively, the System of Panels needs allies and 
this is one way to gain them. Establishing a ranking apparatus requires first 
and foremost that the relevant public views its decisions tis legitimate. 
Having organizations select panel members is tisking them to invest in the 
system. An experienced foundation person commented when being 
questioned about the panels the philanthropy set up to scan its field of 
interest, “We always see that the right and left are represented on the panels”. 
If directors of one of the wealthiest of American foundations felt the need 
to do that while awarding grants without statutory constraints or oversight, 
it might be worth considering in the present case. 

• In addition to the experience of the working sessions, panel members will be 
influenced by the reception given their efforts. The operations of the panel 
could include several follow up efforts that not only highlight the 
designations made but applaud those who arrived at them. OERI 
representatives have been consistently generous in their appreciation of 
participation. This could be extended to include a “day inside the 
Department of Education.” After the first round, would the Secretary meet 
■with the panel and representatives of the projects? Could this be followed 
by shado'wing of DE professionals by panel and project members? NYC 
principals have learned perhaps to their surprise that allo'wing foundation 
representatives into their environment beyond the usual dog and pony show 
yields affirmative results. Also, off the record sessions between civil 
servants and “outsiders” have a long and distinguished history. (Israeli Jews 
and Palestinians met for years under the auspices of Prof Herb Kelman of 
Harvard before either could acknowledge such radicalism publicly). Even 
panel members who are highly conversant 'with the Department’s policies and 
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history might welcome this kind of welcome and deference. 



IV. Disseminating the conclusions of the panels 

Good communications, like good fundraising, goes on all the time. It is best not assigned 
a narrow slot in the cycle to this task. As important as the announcement of the conclusions 
of the Panels will be, it would be desirable to have an on-going dissemination strategy: 

• Establish links back to all participants in the process as it moves forward. 
High on the list would be the organizations who nominated entrants (or were 
approached to do so). They could receive brief tag line updates, e.g., "The 

Panel on will be meeting on April 5* to consider projects ranging from 

a grassroots Haitian grandmothers mentoring effort to the twenty site school 
to work effort of a major medical center.... we thank you again for your 
interest in this pioneering effort to share what works for the covmtry’s 
children." Such brief updates could also include reference to past designates 
when relevant; e.g., “ Susan Murphy, the RN working on 1997 Example 

project will bring her experience and high standards to the panel 

during 1998 when, as it happens, 1/3 of the finalist projects concern the 
provision of medical or mental health services in the schools.” 

• Another aspect of utilizing all connections, would be discussions of the 
approach of the panels themselves. Results will be more carefully absorbed 
if people know of the mandate of the EPs. If funding decisions are to be 
pegged to Panel conclusions, then this could be posted in the various funding 
newsletters and publications with anticipated dates for announcement of 
results, well in advance. The American Association of Fvmdraising 
Executives might view the implementation of the Panels as a newsworthy 
story for one of their national or regional meetings if the process can be 
convincingly shown to be different from previous search operations. 

• Dissemination starts , as far as the wider educational community is concerned 
with the applications for designation. Their characteristics might provide a 
story in themselves. What are some of the most intriguing answers to the “In 
five years . . . .” Question? (page 3 .) What are the results of the attempts to 
offer technical assistance via the buddy system? (page 4.). 

In summary, the organizers of the Expert Panels have responsibility for carrying out several 
major tasks but progress has already been made on several of them. OERI has drawn 
professionals to its vision of what needs to be done and gained the commitment and support 
of the initial panel members. What is clear from any consideration of steps to be taken next, 
is the central role of long term involvement of very able staff. It is hoped that these 
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suggestions may be modestly helpful to those who “do the work.” appreciate having been 
asked to comment. 
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APPENDIX ONE 

Foundation Activities Relevant to the Expert Panels 

1. Foundations have supported particular individuals or research organizations whose 
members may serve on the panels. Examples would be foundation support for Replication 
and Program Strategies or Human Interaction Research Institute headed by Tom Backer. 

2. Foundation support organizations which, while not in the past specifically focused on 
work similar to that of the panels, do have much experience which parallels such activity. 
The Carnegie Council on Early Adolescent Development or the Foundation for Child 
Development would be promising sources for interest in the panels’ findings. 

3. Foundation groupings - some titled affinity groups by the Council of Foundations and 
some not - would be interested in the EP’s work especially as it applies to their interest area, 
(e.g. Youth Development Funders.) 

4. Foundations have a major interest and concern in evaluation. This shows increasingly in 
the operations of inter-foundation exchange. 

The Foundation Center’s Grants Classification System, greatly upgraded and expanded in 
the last five years, now contains a subfield on evaluation making possible findings on 
foundations’ practice. For information: Research Department, Foundation Center, 79 5* 
Ave.,NY, NY (212) 620-4230. 

The annual Foundation Management Survey conducted by the Council on Foundations now 
contains questions on members’ evaluation practice. For information : Judith Kroll Director 
of Research, Council on Foundation, 1828 L St., NW, Wash DC 20036 (202) 467-0432 
krolj@coforg 

The Grantmakers Evaluation Network is a membership organization of about 200 foundation 
professionals which conducts workshops and advocates for evaluation among its members 
and others. For information: Dr. Barbara Kehrer, Marin Community Foundation (415) 461- 
3333 Fax (415) 461-3386 

The Council on Foundations, in cooperation with GEN, will sponsor an extended conference on 
evaluation in November, 1997. Such sessions in the past have drawn sizable numbers of 
foundations interested in expanding their commitment in this area. For information Tara 
Sandercock, Council on Foundations, (202) 467-0432 
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APPENDIX TWO 



The Operations of Award Juries and Possible Parallels to OERI Expert Panels on Promising and 
Exemplary Projects and Products Award programs share some needs with the emerging Expert 
Panels of OERI: they must , to be successful, use members’ contributions effectively, cover a great 
deal of material in a truncated amount of time and make often difficult decisions among 
submissions. 

1. As suggested earlier in this paper, if the demarcation and management of time is placed in the 
hands of staff rather then with a panel member, part of the strain of a day long meeting is 
alleviated. In addition, care needs to be taken concerning the assignments of Panel Members (or 
jurors) before they arrive at a meeting. 

2. Preoccupied Members can be expected to focus better if mailed a scoring sheet (one page, color 
■ coded) and given a very clear set of check off slots among which to sort the submissions . 

(Note, however, that some members won’t involve themselves in this kind of activity; don’t 
become anguished over this. The trick is to induce participation in different ways.) Depending 
on the number of submissions, it may be well to not send the full application to the Panel but 
rather to provide a first, summary sheet (entrants should be furnished the format as part of the 
application packet). 

3. Another widely used approach is to assign each member a certain number of submissions for 
review ahead of time. Given that he or she will anticipate reporting on their four or five to their 
colleagues, this approach encourages preparation. 

4. Preparatory thought given to the setting, logistics and possible alterations of the day long decision 
making day will be well rewarded. The following is an example of a tested decision making 
format. 

9 a.m. Coffee, informal exchange among Panel Members. Staff and Chair make introductions of 
those unknown to each other. 

9:30 to 9:45 Staff outlines the task and procedures to be followed for the day. Clearly noted would 
be those one or two points during the session when Panel Members may , if they wish, alter the 
process, (e.g., returning a discarded entity). 

9:45 to 12 Panel members report on “their” submissions which have been mailed to them in full 
several weeks ahead of the meeting. Fellow members would have cover sheet so that the application 
is not completely unknown to them. Members would have excluded themselves from review of 
those projects with which they had some association. The whole panel questions and debates the 
entry, reviewing any visual material available. A decision is reached whether the particular 
application is to be held in the pool of possible designates. Any panel member would later have an 
opportunity to “bring back” an entrant that had been dropped during the morning session. 
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NOTE: If the field is large, the panel can usefully be divided into “first cut” teams of 2 reviewers. 
The team should preferably be from different disciplines and not have known each other previously. 

12:30 to 2 The Panel addresses the winnowed group of applications, seeking to further narrow the 
field. The goal is to have all panel members reach consensus on those selected for designation. In 
the case of the Expert Panel, rather than limiting the outcome to a given number, the task will be to 
place submissions in Exemplary, or Promising or undesignated. The process should give Panel 
Members time to weigh their decisions while inexorably moving them towards making such choices. 
Techniques to accomplish this include the use of colored dots to be placed by Members on a 
newsprint listing of the (declining) number of submissions in the first category. 

2:15 to 2:45 Panel Members are invited to bring back for reconsideration any rejected entity. 

3 to 3:30 Final consensus is reached. 

3:45 to 4:15 Panel members review critically the process followed and make suggestions for 
conducting the next round. If their participation has been rewarding, they may well volunteer to 
advocate for the Panel in concrete ways within their own constituencies. Such indications of 
outreach are extremely valuable but will require follow up by staff. Panel participation can be very 
energizing but its impact fades when one is back in one's regular life. Again, skilled staff is critical. 

4: 1 5 to 4:30 Panel shares coffee, stimulant, shared congratulations before heading for the airport. 



Other Key Issues for Planning a System of Expert Panels: Making a Complex 

System Viable, Inclusive, and Fair 

Floraline I. Stevens 
Floraline I. Stevens and Associates 



Introduction 

Under its 1994 Reauthorization (Title IX of Goals 2000: Educate America Act), the Office of 
Educational Research and Improvement (OERI) was given the responsibility of recognizing 
promising and exemplary products, programs and practices. Part of the recognition process involves 
the following: (1) developing standards for recognizing promising and exemplary products, 
programs and practices; (2) establishing a system of expert panels to make recommendations on the 
designations to the Secretary of Education; and (3) coordinating dissemination activities and 
programs within OERI, other parts of the Department of Education, and other agencies concerned 
with education such as state education agencies and local school districts. 

Purpose 

The purpose of the expert panels is to enable the federal government to help the public learn 
about the comparative advantages of what exists among the available R&D-based products, 
programs and practices; and to help the federal government and other funders and producers of 
R&D-based resources maximize their investments by providing support for further evaluation, 
improvement and dissemination. The Department wants to design a dissemination system that 
makes programs available to the public as quickly as possible, to respond to different kinds of 
requests, and to support the application of research and best practice. 

Background 

Two pilot expert panels on mathematics and science and on gender equity were established in 1996. 
They worked on the issues of process and substance. In 1997/1998, two additional expert panels on 
technology and on early reading will be established. Panelists must be individuals who have 
in-depth knowledge of the subject area or content of the program or group of programs to be 
evaluated. At least one current teacher, principal, or other school-based or community-based 
professional must be on each panel and no more than one-third Federal employees. 

A standing panel of experts and practitioners from across fields will coordinate and monitor the work 
of the expert panels. The standing panel members must have demonstrated expertise and experience 
in at least two of these qualifications: 

• specific educational areas. 

• broad range of educational policies and practices. 

• evaluating education programs 

• product development and/or dissemination 



current employment as a teacher, principal or other school-based or 
community-based professional. 



Submissions 

Any public or private agency, or organization or institution, or an individual may submit an 
educational program to the System of Expert Panels for review. The submission will contain a 
description of the program, program materials, and a discussion of the program's evidence of 
effectiveness or success, quality, significance, and usefulness to others. The latter are the criteria 
forjudging whether or not a product, program, and practice is promising or exemplary. 

Issues Identified from the First Year Formative Evaluation: Pilot Expert Panel in Gender Equity 
Campbell (1996) identified several key issues that need to be addressed in planning for the 
implementation of a System of Expert Panels. 

They were : 

• Overdependence on volunteers. Panelists are not compensated for their time or expenses, 
panelists are limited in the amount of panel work they can undertake, particularly 
panel-related correspondence. 

• Perceptions of conflict of interest or bias. There are a variety of conflict of interest or 
bias issues at work, including relationships that panel members have with developers, 
the developer status of some panel members, and the concerns that there were political 
criteria in terms of panelist selection. 

• Lack of clarity on what the process is. Panel members are not clear about the steps to 
be taken towards designating programs. There was little agreement among panelists on 
exactly what the submission process was. 

• Lack of clarity on the benefits of going through the process. Many panelists were not 
sure what developers and others would gain from submitting programs to the panels. 

• Lack of common definitions for such key concepts as promising and exemplary and the 
nature of evidence. It appears that common definitions for these and other terms need 
to be better defined and consensus between panel and sub panel members reached before 
products are reviewed. 

• Lack of a comprehensive program to get submissions. Panel members are not clear 
about how to get programs to submit to the panel and there is little agreement among 
panelists what the submission process was and who is responsible for what aspects of it. 

• Funding. In order for the panel to continue its activities, sustained federal funding for 
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activities must be found, including money for panelists' honorarium and expenses, 
administrative support, and dissemination of submissions approved as either promising 
or exemplary by the panel. 



Other Planning Issues to Be Addressed 

The remainder of this paper will attempt to identify other issues that need attention in planning a 
system of panel experts. Patricia Campbell in her formative evaluation report and Lois-ellin Datta 
in her paper on incorporating research findings and practices have addressed some of these issues 
but in different contexts. These are the three areas that will be discussed: 

• Promoting an organizational structure that is manageable. 

• Having outreach that is inclusive versus exclusive. 

• Ensuring that selection decisions and designations are fair. 



Organizational Structure 

It is laudable that the Department wants its role to be facilitative but merely being facilitative is not 
the opposite of demonstrated administrative leadership and responsibility. In this respect, there is 
concern about the current suggested configuration called The Wheel Diagram of the Key Elements 
of the System of Expert Panels. 



The Wheel Diagram of the Key Elements of the System of Expert Panels. This diagram graphically 
depicts a rather complex and complicated system of organizational structure. There is a legitimate 
question of whether or not this type of structure will work when multiple panels are selected and 
operating and the number of submissions increase. The first issue concerns the Federal Coordination 
Unit (FCU) of experts. As described, this unit is the hub that will be responsible for the coordination 
and leadership for the entire system. Since the participants will be borrowed firom other agencies, 
there is the question of enough time allowed to do this unit's work and the ability to commit to do 
this work in addition t o regular Department of Education responsibilities. Second, what staff person 
will be responsible for the leadership of the FCU. It cannot be a lower-level person when 
coordination among the agencies is a must. For this coordination and leadership to occur, this 
appears to be a full-time job not supplemental to other tasks. Third, there is no description of how 
the standing panel is different firom the expert panels and how this standing panel relates to the 
semi-autonomous topic-focused expert panels and to the FCU. Estabhshing a platform for workable 
and positive interrelationships is a critical issue that needs to be addressed. 



Other Organizational Structures Using Experts. Looking at the experts' organizational 
structure for The United States Food and Drug Administration (FDA) and the coordination of its 
multi-facet work may be helpful in reducing the complexity of the System of Expert Panels' 
suggested organizational structure. The FDA has a Center for Food safety and Applied 
Nutrition (CFSAN). This center has one chartered advisory committee. This is a technical and 
scientific committee that advises the Center and agency on emerging food safety and food 
science and nutrition issues. The committee consists of 1 8 standing members with expertise in 
the physical sciences, biological and life sciences, food science, risk assessment, or other 
relevant scientific and technical disciplines. Most members are drawn from academia, 
government, and professional societies. Some technically qualified members also represent 
consumer and industry interests. As necessary, temporary members are also appointed to provide 
additional expertise for consideration of specific issues. 

Another expert organization structure exists within the World Technology Evaluation Center 
(WTEC). The Center is housed at Loyola College and is funded by a cooperative agreement from 
the National Science Foundation (NSF) to perform a series of international technology assessments. 
Panels of experts under the WTEC methodology assess international R&D in a number of 
technology areas. Small panels of about six technical experts conduct WTEC assessments. Panelists 
are leading authorities in their field, technically active, and knowledgeable about research programs. 
As part of the assessment process, panels visit and carry out extensive discussions with scientists and 
engineers in universities and in industry or government labs. 

The staff at Loyola College help select topics, recruit expert panelists, arrange study visits to 
laboratories, organize workshop presentations, and finally, edit and disseminate the final reports. 

Artificial Intelligence As a Means of Working Smart 

If the submissions of products, programs and practices grow geometrically, there is the 
possibility that the system will collapse from the complexity of its own weight. One desired 
outcome for the system is that there is timely dissemination of information about promising and 
exemplary products, programs and practices to educational consumers and the public. To address 
the issue of timeliness as well as reducing the need for more human resources is to consider 
using expert systems from artificial intelligence as a means to initially process parts of the 
submissions. A computer generated expert system uses a computer program that behaves like a 
human expert in some useful ways. Structural components of the system are: 

• Use progranmiing techniques of artificial intelligence for problem solving. 

• Use a knowledge component within a computer. The knowledge of an expert 

system consists of facts and heuristics (little rules of good judgment or 

plausible reasoning that characterize expert-level decision making in the 
field). The "facts" constitute a body of knowledge that is widely shared, 
publicly available, and generally agreed upon by experts in the field. 



Currently, there is a strong dependency upon volunteer work. Campbell's (1996) formative 
evaluation report suggests that this is unrealistic if the work is continuous, steady, and increasing. 
In addition, it is contrary to the organizational structure of other groups when experts are called 
upon to share their expertise. Experts are typically paid at a rate comparable to the income they earn 
from other assignments for evaluating a case, preparing a report or giving testimony. Artificial 
intelligence is a means to reduce the time needed for human experts to assess submissions and may 
make it more fiscally feasible to pay human experts for their expertise. 

Inclusive Outreach. 

At the 1997 AERA meeting, it was evident that OERI is already concerned that the process of 
submissions is inclusive rather than exclusion. This is the importance of having a public 
relations/dissemination plan before the System of Expert Panels becomes fully operational. When 
working on a dissemination advisory committee for the Beginning Teacher Study, months of 
discussion and planning culminated in a successful outlay of information to the media print, voice 
and electronic about the study. If the Department wants to move beyond the professional and 
advocacy groups to include others, the outreach must be systematic and not ad hoc. The whole idea 
of expert panels needs to be fully communicated to all that should hear about it. Certainly there was 
a beginning at AERA in Chicago. However, that was the "in-crowd". The popular means of 
communicating with teachers, principals and the public should be tapped. To be considered is the 
Phi Delta Kappan, ASCD publications. Education Week, AERA journals, topic and subject journals, 
and the education reporters at local and national newspapers. In addition, attending conferences and 
meetings of professional groups, parents groups, and others is needed to get the information out and 
discussed. The internet must be included too for those who have computers. Schools of Education 
must be contacted so that their faculties who are involved in educational research will become 
informed. There are other avenues not listed but a dissemination/communication advisory 
committee could get the process of inclusive outreach started. 

Selection Decisions and Designations That Are Fair. 

Within the concept of a consumer-related evaluation (Slavin, 1996), there must be an assessment 
process that is fair and is viewed as fair by education consumers and the public. If decisions are 
made which inform educational consumers (teachers, principals, support staff, program developers, 
and the public) about the merit of an educational product, program or practice, these decisions must 
be based upon valid and reliable information. 

As was previously stated, judging criteria will be used by the expert panels to distinguish between 
products, programs and practices that are promising or exemplary. Not mentioned but part of the 
process are the submissions that did not meet the cut. What kinds of feedback are needed by the 
group whose submissions are not accepted?. It is an omission that needs attention because, once the 
system is established, there will probably be many products, programs, and practices in the "below 
the cut" category. To be fair, the "cut decisions" need to be described in quite specific detail and 
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terminology. It is discouraging to receive rejection information that is vague and non-informative 
no reasons provided for the rejection. Constructive feedback leads to improved planning and 
implementation for new products, programs and practices. 

For those products, programs and practices that are accepted, the process for determining 
designations of promising or exemplary must be valid and reliable. If not, the integrity of the system 
will be in question. Validity and reliability are particularly important because the designations of 
promising and exemplary will be public and not private. 

Presently, the standards/criteria are the following: 

• Evidence of effectiveness/success 

A strong program must have defensible overall evidence supporting claims of 
worthwhile performance results at one or more sites (without failing at a large 
number of other sites). The program must have logical or other evidence of 
adaptability and transportability to other sites. An excellent program must 
meet all of the above criteria and have very important claims of positive 
results (performance outcomes) that are attributable to the program and 
evidence that supports these claims of worthwhile results that were sustained 
in multiple site replications within the past few years. 

• Quality 

The program: (a) is congruent with sound research and practice; (b) incorporates 
accurate and up-to-date information/content; (c) promotes equity and is free of 
bias based on race, gender, age, culture, ethnic origin, disability, or limited 
English proficiency; (d) is appropriate, engaging, and motivating for the intended 
audiences; and (e) contains materials that conform to accepted standards of 
technical product quality. 

• Educational significance 

The program addresses an important education issue, challenge or problem 
and has advantages over other programs with similar purposes. 

• Usefulness to Others 

The program is: (a) reasonable in terms of costs to potential users in relation 
to expected benefits; (b) is or can be made easily available to potential users; 
(c) can be readily adopted or easily adapted in new locations; and (d) can be 
used in conjunction with other programs, if appropriate. 
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Using A Portfolio System to Assess the Standards/Criteria. 

The Gender Equity Expert Panel in 1996 extended an invitation for submissions of programs and 
products that had immediate application to increasing gender equity in schools, colleges and 
universities, and community organizations. The types of materials suggested were print or non-print 
items, including multimedia formats such as software, videos, or audiotapes. Also suggested for 
inclusion were formal evaluations or longitudinal studies, participant evaluations, reviews in local 
media, or indications of shifts in thinking or behaviors. All of these items or combinations of items 
in essence became a portfolio of evidence to show the impact or success of the program or product. 
How do you assess a portfolio of materials to determine whether standards are met? The New 
Standards Program at the University of Pittsburgh is working on this same question. 

The New Standards Program incorporates performance descriptions of what students should know 
and the ways they should demonstrate the knowledge of skills acquired. Work samples and 
commentaries are used to illustrate the meaning of a performance standard together with 
commentary that shows how the performance descriptions are reflected in the work sample. The 
portfolio system provides example portfolios that contain concrete examples of expectations for 
students and teachers so that the standards are clear and usable. 

The System of Expert Panels' submissions can include a similar process. A portfolio assessment 
approach could be used to extend the definitions of the standards forjudging and assessing products, 
programs and practices. This standardized approach helps to insure inter-rater reliability among the 
expert panelists. 

Inter-rater reliability 

Are there to be points assigned to the different levels of meeting these criteria? Without this 
happening, human judgments can be scattered over a wide range of possibilities. Inter-rater 
reliability among expert panelists is an important issue to be addressed. Descriptions of each 
criterion should be thorough and comprehensive enough so that each expert is rating the same 
thing in the same way. Reliable assessment means that there is consistency in how a product, 
program or practice is judged. Inter-rater reliability means that there is consistency among the 
raters in how they rate or judge a product, program or practice. Scriven (1991) notes that 
calibration exercises, checklists and training of the raters help to increase reliability. The advice 
of assessment experts is needed to address the issue of reliability and validity. 

Validity 

In the assessment context, there are several forms of validity face, content, concurrent, construct, 
and predictive. Valid evaluations or assessments of products, programs and practices must take 
into account all of the relevant factors. In the context of submissions to the expert panels, the 
criteria would be the relevant factors. However, what are the weights given to the four criteria or 
are they equal in importance. Slavin(1996) notes that educational consumers view improved 
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educational outcomes for students improved scores, grades attendance, attitudes and others — as 
most important. Does this mean that evidence of effectiveness or success should have greater 
weight than the other three? This is certainly something that needs to be discussed and decided 
upon. 

Clear and concise definitions of promising and exemplary are critical to making judgment decisions 
about products, programs and practices. These definitions are needed to train raters in various 
calibration exercises So that their decisions are valid and reliable. 

Reliability is a replicability issue. 

Consumers want to be sure that the product, program and practice will produce similar results if 
implemented according to the descriptions and guidelines. Consumers want to know if the original 
results are context-bound or whether the product, program or practice would be as effective or 
successful in a different context. Assessment persons would have to develop a fimnework for 
assessing the reliability and replicability for these areas. This issue of replicability is mentioned in 
the Evidence of Effectiveness/Success criteria standard. However, it was not quite clear what 
"logical evidence or evidence of adaptability or transportability" really mean. These terms need to 
be better defined and given a quantifiable scale. Several alternative pilot assessments of 
submissions need to be conducted by the expert panels to ascertain what processes are best to make 
reliable jugdgments. Also, expert panelists with their wealth of knowledge and skills need to 
practice them in the context of the work they are responsible for doing. No matter how expert a 
panel may be, there continues to be a need for training and time for reflection about the related 
information before the process of review/judging is undertaken. This is particularly important 
because of the composition of the panels automatically impose diversity of skills and approaches and 
only training brings about some consensus to the assessment tasks. 



This paper suggests that there are other issues that need to be addressed when planning a System 
of Expert Panels. These include: 



Organizational structure must become less complex than what is presently proposed because as the 
panels and submissions increase, the proposed stmcture may be burdensome and difficult to manage. 
The FDA's Center for Food Safety and Applied Nutrition as well as the World Technology 
Evaluation Center at Loyola College have expert systems that are successful operations. Another 
consideration is the use of a computer-generated expert system to assist in the initial assessment of 
products, programs and practices. 



Summary 



(a) promoting an organizational stmcture that is manageable; 

(b) having outreach that is inclusive versus exclusive; and 

(c) ensuring that selection decisions and designations are fair. 
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Outreach is another issue to be addressed because OERI does not want the system to be viewed as 
elitist or exclusive. Pre-implementation planning by a dissemination advisory committee is 
suggested as a means to get information out to multiple audiences. 

The last issue is the need for education consumers and the public to feel that the selection and 
designation process is fair. It is suggested that the system borrow from assessment/testing practices 
to assure validity and reliability of decision-making. Also, to handle the complexity of multiple 
materials in a submission, a portfolio system with concrete examples and accompanying narrative 
descriptions would enhance the clarity of performance expectations and increase inter-rater 
reliability. This information will also impact the replicability of the product, program or practice. 
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Expert Panels and Educational Reform: 
Some Perceptions Based on 
a Search for Promising and Exemplary 
Adolescent Pregnancy Programs 

Karen Bogart, Ph.D. 

Anne Steinmann Institute and EduTech Limited 



The OERI Reauthorization of the Goals 2000; Educate America Act of 1994 calls for the formation 
of expert panels to judge the success of innovative educational programs, policies, practices and 
products. Those programs and products designated promising or exemplary are to be disseminated 
by the U.S. Department of Education. The intent of the legislation is to provide a federal stamp of 
approval based on expert review that will encourage schools and communities across the nation to 
replicate outstanding programs and by so doing, improve educational opportunities and outcomes 
for students nationwide. As a tool for motivating and directing educational reform, the Expert 
Panels has provoked considerable discussion and differences of opinion among the educational 
practitioners, researchers, and evaluators who have participated in the first two pilot panels, launched 
early in 1996, and/or reflected on them in commissioned papers written between 1994 and 1997. 
The dialogue engendered by the Expert Panels has addressed such issues as: 

• Basic philosophical issues, including the extent to which the use of expert panels can 
promote widespread educational reform. This translates into an additional set of sub-issues, 
including (1) whether replication of discrete programs, policies and practices by a single 
school or perhaps a school district, such as an exemplary reading program from one source, 
an exemplary physics program from another, and an exemplary educational technology 
program from yet another, will promote systemic educational reform; (2) whether schools, 
commimities and states will be motivated to replicate programs that have received U.S. 
Department of Education recognition as promising or exemplary; and (3) whether and how 
the Department of Education can most effectively disseminate information to schools, 
communities and states, (i.e., by what combination of online and hard copy approaches.) 

• Whether the Expert Panels will, in fact, be able to recommend programs, policies, practices 
and products that truly meet their own criteria of success or effectiveness, quality of 
program, educational significance, and usefulness to others, or whether they will, in effect, 
be recommending programs and products for which the evidence is marginal at best, 
especially in relation to the criterion of success or effectiveness. The concern is based, first 
of all, on the expectation that many, if not most, innovative programs, policies, practices lack 
the necessary formative and summative evaluations (especially, third party evaluations) 
needed to fine tune them in the developmental stage (formative evaluation) and assess their 
impact once they are developed (summative evaluation). There has also been a concern 
articulated that the less knowledgeable Expert Panel members are about evaluation, the more 
rigid they will be about demanding only one approach to evaluation — an approach 
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involving control groups and quantitative criteria such as test scores — although evaluations 
of complex social and educational programs now take many different forms and include 
qualitative as well as quantitative evidence. 

• Whether programs, policies, practices and products should all be held up to the same four 
standards. The concern here is that while programs can be asked to meet criteria of success 
or effectiveness as measured, for example, by improved learning as shown by test scores or 
other means, it is much more difficult for products such as textbooks or videos, in particular, 
to meet this criterion. 

• What the composition of the expert panels should be, including the extent to which panel 
members should be researchers and evaluators first vs. whether they should not be 
researchers or evaluators, although they should have some knowledge of research and 
evaluation. 

• Whether the structure and dynamics of the Expert Panels should be tightly or loosely 
controlled, with formal meetings at regular intervals, and a reasonable budget, including 
payments for panel members and reviewers, in order to ensure both (1) that panel members 
and reviewers can make the commitment, invest the time, and perform their work to the best 
of their abilities and (2) that the work they do is taken seriously by the educational 
commimity at large. 

• Whether each panel’s efforts should begin with a research synthesis that reviews the state of 
the art in the field. 

• How program directors can be encouraged to go through the process of submitting their 
programs, policies, practices and products for consideration, when the submission process 
is formidable, and the benefits of submission are not clear. 

• How the programs, policies, practices and products are to be disseminated. 

As a researcher and evaluator who is now part of an effort sponsored by OERI to collect information 
about school-based and school-linked programs that address the educational, economic, social and 
health needs of pregnant and parenting adolescents, these observations are of special interest. This 
paper examines the questions asked by commissioned papers authors in the light of our own recent 
experiences. The remainder of this paper outlines the approach that we are using to identify 
adolescent pregnancy programs we hope will meet promising and exemplary programs, as these have 
been defined by OERI. 
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Method 



Our approach includes: 

• Boolean searches of various general Internet databases, using Alta Vista, Yahoo!, Excite. 
Boolean searches of various government databases, including those of (1) the Department 
of Education and (2) the Department of Health and Human Services. The purpose of these 
searches has been to identify articles, books, existing resource compendia to model 
adolescent pregnancy programs as well as national associations and non-profits that serve 
them, statistical reports, white papers, speeches, recent legislation and policy issues 
(especially related to welfare reform). 

• A review of the activities of foundations that provide funding to adolescent pregnancy 
programs (e.g., the Stuart Foundation in San Francisco, CA). 

• Contacts by mail and by telephone to candidate school-based and school-linked programs 
identified through these sources, as well as to knowledgeable individuals and organizations 
for information about other outstanding programs that may not be described in any 
literature. Inquiries by mail (1) ask the program to reply if it would like to be considered 
for inclusion in an OERI-sponsored resource guide to promising and exemplary school- 
based and school-linked adolescent pregnancy programs and (2) seek more detailed 
information about the program. The information asked for includes basic demographics, 
the populations served by the program, the locus of services (e.g., school, hospital, 
home), the kinds of services offered, and whether the program can offer evidence for its 
success in terms of specific performance indicators. (See Appendix A for project brief and 
Appendix B for guidelines mailed to programs). 

• Discussions by telephone and (if possible) in person with the principals of the program, 
for the purpose of obtaining more precise information about the program that could be 
used in a description and, especially, about the evidence that the program can offer that 
it meets criteria established for classifying programs as promising or exemplary. 

• Discussions with nationally recognized experts in the field as well as with principals, 
teachers, and adolescent parents themselves in an effort to represent critical needs that 
must be addressed if adolescent pregnancy programs are to succeed. A primary objective 
of discussions with experts is prevention of low birth weight. 

• Advice and direction from a distinguished advisory panel that may be asked to review and 
make final recommendations on candidate programs that meet preliminary criteria of 
promising and exemplary programs. 

We have to date contacted more than 400 knowledgeable individuals and organizations seeking 

nominations of promising and exemplary school-based and school-linked adolescent pregnancy 
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programs. Each contact has been provided with a project brief and with guidelines for replying 
about the populations served by each nominated program, the services provided, the locus of these 
services (e.g., school, hospital, home), the evidence the program had for its success, and whether 
the program has been replicated. 

More specifically, we have contacted... 

• 131 school-based and school-linked adolescent pregnancy programs described in various 
resource compendia including: 

• Adolescent Pregnancy Prevention: A Compendium of Programs published by the 
Healthy Mothers, Healthy Babies Coalition (1995), associated with the American 
Association of Obstetricians and Gynecologists; 

• The Adolescent Family Life Demonstration. Projects: Program and Evaluation 
Summaries published by the Office of Adolescent Pregnancy Programs (1990); 

• School-Based Programs for Adolescent Parents and their Young Children: 
Overcoming Barriers and Challenges to Implementing Comprehensive School- 
Based Services published by the Center for Assessment and Policy Development 
(CAPD) (1994); 

• School-Based Programs for Adolescent Parents and Their Young Children: 
Guidelines for Quality and Best Practice published by the Center for Assessment 
and Policy Development (1996). 

• 100 directors of national associations and other non-profits identified in Adolescent 
Pregnancy Prevention: A Compendium of Programs, 

• 50 state liaisons to the National Organization of Adolescent Pregnancy, Parenting and 
Prevention Programs. 

• OERI’s Gender Equity Expert Panel ListServ of more than 100 educational researchers 
and practitioners. 

The adolescent health programs outlined in an additional compendium of the Maternal and Child 
Health Bureau of the Health Resources and Services Administration (HRSA) were also reviewed 
for this report. We are currently seeking further information about 40 of these programs, 
including a number that are Healthy Start programs, since these have a primary focus on reducing 
low birth weight and improving outcomes of infants bom to adolescent parents. Although these 
programs are not as clearly school-linked, they may be critically important to reducing low birth 
weight in the 50% of teen mothers who drop out of school. No evaluative data is as yet available 
on these currently funded programs. The programs are described in: 
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• Adolescent Health Report published by the Maternal and Child Health Bureau, Health 
Resources and Services Administration (1996). 

In addition, we have conducted conversations with key individuals in an effort to ascertain the 
extent to which the available resource materials describe best practices that accurately and 
completely address health, educational and social needs of pregnant and parenting adolescents. 

Preliminary Results: 

Promising And Exemplary Programs 

Our initial outreach efforts haven been successful in producing nominations of more than 70 school 
based and school linked adolescent pregnancy programs. More than 59 provided information in 
enough detail for us to make a preliminaiy judgment about whether to seek further information about 
them. Currently, as we review candidate programs for consideration, we are guided both by OERI’s 
work to date establishing four broad criteria for promising and exemplary programs and by the 
approach of educational practitioners and researchers who identify four basic performance indicators 
for assessing the success of school based and school linked adolescent pregnancy programs. The 
interfacing of OERI criteria and the performance indicators commonly used to assess the outcomes 
of adolescent pregnancy programs yields Table 1. 
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Table 1 

Interfacing of OERI Criteria 
for Promising and Exemplary Programs 
with Performance Indicators Used to Measure Outcomes 
of Adolescent Pregnancy Programs 



Success/Effectiveness 
Performance Indicators 

Reducing Drop Out Rate 

Qualitative: Students report that the program enables them to remain in school or that they are 
drop outs who returned because of the program. 

Quantitative: Drop out rate of students in program is lower than that of comparison groups, such 
as students in other programs or in community as a whole. 

Reducing Low Birth Weight/Improving Health And Well Being Of Children Born To 
Adolescent Parents 

Qualitative: Students report that they obtained prenatal care, paid attention to nutrition, avoided 
substance abuse, avoided domestic violence. Students report participating in health education and 
parent education courses. Students report that their children are in early childhood education 
programs. Students provide evidence that program has helped them be better parents who 
understand early childhood development, avoid child neglect and abuse, etc. 

Quantitative: Birth weights of infants bom to students in the program are higher than those of 
comparison groups, such as students in other programs or in the community as a whole. 

Reducing Repeat Pregnancy 

Qualitative: Students report that they are taking actions to avoid pregnancy. 

Quantitative: Students perform well on performance indicators, with fewer repeat pregnancies 
reported than in comparison groups. Siblings of adolescent mothers — at very high risk of a first 
pregnancy — report fewer pregnancies than comparison groups. 

Improving Economic Self-Sufficiency 

Qualitative: students report that they are continuing and completing their education, obtaining 
postsecondary education including vocational training and college educations, and finding, 
keeping and being promoted in employment. Graduates include true success stories — students 
who have gone on to careers in the professions, including medicine, law and teaching, and who 
may now be helping other adolescent parents. 

Quantitative: students perform better than comparison groups in terms of numbers who graduate, 
obtain employment, receive further vocational training, and obtain college education. 
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Table 1 (Continued) 



Quality of Program 

Program Activities That Mediate Outcomes 
Reduces Drop Out Rate 

Outreach to pregnant and parenting adolescents in community: program undertakes specific 
activities to inform pregnant adolescents, their parents, teachers, principals, health care providers, 
social service providers of its services - does not depend totally on initiative and motivation of 
pregnant adolescent to learn about the program and apply; 

Program provides case workers who work with students on one-to one basis, including if needed 
home visits; 

Program offers flexible scheduling - evening, weekend classes, summer classes, opportunities for 
making up missed work, e.g., using computer based instruction; 

Program provides on-site child care; 

Program offers one stop shopping — all services at a single site — a holistic approach; 

Reduces Low Birth Weight/Improves Health And Well Being Of Children Born To 
Adolescent Parents 

Health education, HIV/AIDS education, on-site child care, early childhood education. 

Program serves pregnant minors, whether or not they have dropped out of school (they may 
always return later). 

Health care and prenatal care are provided by part time or full time on site physicians, nurse 
practitioners and other health practitioners, preferably with training in pediatrics and adolescent 
medicine. 

Reduces Repeat Pregnancy 

Health education, parenting education courses. 

Peer counseling. 

Mentoring. 

Programs for siblings of teen parents. 

Improves Economic Self-Sufficiency 
Academic courses. 

Vocational courses 

Use of technology as both end (learning technology) and means to end (e.g., using computer to 
remediate basic skills in language and math). 

Academic counseling. 

Vocational counseling. 



Table 1 (Continued) 



Educational Signifigance 



All Indicators 

Addresses basic humane and societal needs. 

Has unique characteristics that merit emulation by others. 
Is as effective or better than existing approaches. 



Table 1 (Continued) 



Usefulness to Others 

All Indicators 

Program has succeeded in a single site. 

Programs have been successfully replicated as measured by success on multiple indicators in 
more than one site. Program activities are not unduly complex nor restricted to the conditions 
and facilities associated with a single site. 

Strategies are cost effective, as measured in time, money and numbers of people required to 
replicate. 
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We have used the preliminary data we have received about 59 school-based and school-linked 
adolescent pregnancy programs to make a preliminary assessment of the extent to which these 
programs meet the OERI criteria outlined in table 1 . On the basis of this classification, we have 
differentiated programs into (1) 23 outstanding programs that we expect will, for the most part, 
qualify for designation as exemplary based on successful replication in multiple sites; (2) 18 good 
to very good programs that we expect will qualify for designation as either promising (12 
programs that have not been replicated) or exemplary (6 programs that have been replicated in 
multiple sites), (3) 13 fair programs some of which may ultimately be designated promising and 
(4) 5 indeterminate programs that cannot be assessed because they are too new to have generated 
data measuring their effectivenss. A sample of preliminary descriptions of programs selected for 
follow-up is shown in appendix C. 

Groups One and Two Programs will definitely be followed up with a telephone interview and 
requests for additional supporting data before presentation to our advisory panel for 
recommendations on designation as (1) exemplary, (2) promising or (3) neither promising nor 
exemplary. Groups Three and Four Programs may be followed up, since at least some of these 
programs have innovative features that may be described as potentially promising, even though 
they are too recent to have generated data demonstrating their success even in a single site. We 
can also differentiate a Group Five, comprising curriculum-based pregnancy prevention programs 
that were developed for students who have never experienced a pregnancy, yet may also be very 
useful in preventing repeat pregnancies in parenting adolescents. Some of these programs may 
be followed up as well. 

As shown in Figures 1 through 7 in appendix D, although the most promising of the programs 
were assigned to Groups One or Two on the basis of whether or not they offer early childhood 
education programs for the infants, toddlers and preschoolers bom to teenage mothers, the 
programs in these two groups are actually very different. Group One Programs (N=23) are 
overwhelmingly school-based comprehensive alternative public schools in a position to serve the 
children bom to teen parents with child care and early childhood education programs. Many of 
the Group Two Programs are school linked programs lodged in hospitals or community-based 
nonresidential centers without facilities for child care. A number of the Group Two Programs 
(N = 18), in fact, are current grantees of the Office of Adolescent Pregnancy Programs in the 
Department of Health and Human Services, which further may explain their focus on health care 
and social services . Group One Programs also differ from other programs in that they have been 
in existence for many years. They have not only stood the test of time but also, interestingly, seem 
to be those most able and willing to be innovative, perhaps because they have the resources and 
experience to make innovation work. 

The next phase of our study involves direct contacts with the program director, telephone 
interview and submission of supporting documentation to demonstrate that the program does, in 
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fact, offer the services it claims to offer, serve the numbers it claims to serve, and achieve the 
success it claims to achieve. Few if any programs will be able to offer us the results of third party 
evaluations. However, even if we cannot evaluate a program independently, we will develop some 
means for verifying the authenticity of the reports we receive about programs we are considering 
for inclusion in the Resource Guide and designation as promising or exemplary. We may ask to 
speak with adolescents who have been through a sample of the programs we are considering for 
inclusion in the Resource Guide, with their parents or with school personnel or other personnel 
who are knowledgeable about the program. 

In sum, the best adolescent pregnancy programs are more appropriately described as constellations 
of programs rather than what may often be thought of as a program, such as a curriculum or an 
instructional approach to encourage reading in elementary school age children or improve physics 
education in secondary schools. The most outstanding adolescent pregnancy programs typically 
comprise multiple curricula, instructional approaches and activities often offered in a single site. 
These may include prenatal care; health care for adolescent parents and their infants, toddlers and 
preschoolers; health education and HIV/AIDS education programs; on-site childcare and early 
childhood education programs by certified early childhood teachers; parent education programs; 
academic and vocational education programs; social services including case management, peer 
support groups, mentoring, and programs for extended family members including grandparents and 
siblings. 

A review of these programs addresses, we believe, some of the questions raised by commissioned 
authors. Although we are still at the incipient stages of developing the resource directory, we have 
succeeded in identifying many programs that appear to meet promising and exemplary criteria 
as these have been defined by OERI. Most importantly, to our surprise, many of the programs 
called to our attention identified have been successfully replicated, and not once but over and over 
in different sites locally, state-wide and, in some cases, nationally. They have clearly promoted 
educational reform, even without the benefit of special incentives, expert panels, and national 
recognition or even local publicity. One can only imagine how much greater their impact can be, 
with the added incentives and visibility provided by expert panels, review and recognition at the 
national level and widespread dissemination. A number of promising programs, offering evidence 
of success in a single site, have also been called to our attention. Some of these programs, 
moreover, have clearly succeeded on a large scale, improving education, health and social welfare 
for hundreds of adolescent parents and their infants, toddlers and preschoolers a year. Like 
exemplary programs that can offer evidence of success in multiple sites, these programs deserve 
recognition. Yet others are unique approaches that meet the criteria of educational significance, even 
though they are too new to offer substantial evidence of successful outcome or even though they 
were designed for a somewhat different population, such as adolescents who never experienced a 
pregnancy. These too deserve to be publicized. We believe, in other words, that programs such 
as these demonstrate the soundness of an approach such as OERI’s, that seeks to identify, recognize 
and disseminate successful programs as a means to promoting educational reform. At the same time, 
it is our belief that there are many avenues to educational reform. Expert panels placing the stamp 
of approval on programs that the Department of Education disseminates offers one important avenue 
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for accelerating reform, although it is not the only one. 
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Edutech Limited 



8455 Colesville Road, Silver Spring, MD 20910 
301-585-1030 EDUTECH@ARI.NET 



Project Brief 



School based and school linked programs addressing the educational, economic, social and health needs of 
pregnant and parenting adolescents will be featured in a Resource Guide to be developed by Edutech, an 
educational consulting firm based in Columbia and Silver Spring, MD, under the auspices of the Office of 
Educational Research and Improvement. This publication will encourage replication in other schools and 
communities nationally as well as assist educators, health professionals, social service professionals, 
adolescent parents and their families locally. 

Programs can be based in schools, community centers, health centers, recreational facilities, or even involve 
home visiting. Edutech is especially interested in programs that: 

• Decrease the school dropout rate of adolescent parents 

• Improve the health and well being of children bora to adolescent parents 

• Decrease repeat pregnancies of adolescent parents 

• Increase the economic self-sufficiency of adolescent parents 

Edutech is especially interested in programs that qualify for designation as promising (can provide evidence 
of success in one site) or exemplary (programs that have been replicated successfully in more than one site). 

Please tell us about programs you believe may qualify for inclusion. Programs may address multiple needs 
of pregnant and parenting adolescents or a single need. They may be fairly new or well-established. They 
may focus on adolescent mothers and their children only or on adolescent mothers and fathers. You may 
contact us by postal mail or by e-mail. 

President, Edutech: Dr. Mildred Lockhart Boyd 
Senior Vice President: Dr. Gerald Boyd, Sr. 

Vice President for Business Development: Mr. Gerald Boyd, Jr. 

Director of Research and Evaluation: Dr. Karen Bogart 
Educational Researcher: Ms. Dianne Dale 
Vice President for Administration: Ms. Robin Bailey 
Office Manager: Ms. Kim Jones 
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Edutech Limited 

8455Cole$ville Road, Silver Spring, MD 20910 
301-585-1030 EDUTECH@ARI.NET 



School/Community-Based Program 
For Pregnant And Parenting Adolescents 
Baseline Information Checklist (Form II) 

Your program has been nominated for inclusion in a forthcoming publication of the Office of 
Educational Research and Improvement of the U.S. Department of Education to be disseminated 
nationwide. The first of its kind, this publication will be a Resource Directory to promising and 
exemplary programs in the United States that address the needs of pregnant and parenting 
adolescents. Its intent will be to encourage replication in other schools and co mmuni ties nationally. 

Nominations have been solicited from nationally recognized experts on adolescent pregnancy: in 
schools and school districts; in local, city and state education, health and social service agencies; in 
hospitals, health centers; in colleges and universities; and in private associations devoted to maternal 
and child health, education and welfare. 

Please take a few minutes of your time and tell us about your program. School based and school 
linked programs qualifying for national recognition as promising or exemplary will be featured in 
a Resource Guide to be developed by Edutech, an educational consulting firm based in Silver Spring, 
MD, under the auspices of the Office of Educational Research and Improvement. You may also be 
eligible for federal funding based on your designation as an exemplary program addressing the needs 
for gender equity. 

Please also send copies of brochures, newspaper articles, reports, letters of commendation, awards, 
film clips and any other information on the effectiveness of the program or policy and changes which 
it has helped bring about. The most outstanding programs identified may be invited to participate 
in a telephone interview or receive a site visit that will make it possible to describe your program 
more fully. 



Program 

Programs are asked to provide information such as the following: 



I. 


Where the program is lodged 




Social Services/Programs 






□ 


Case management 


□ 


A public school (is a school-based program within 


□ 


Individual and group personal counseling 




a public school) 


□ 


Programs for adolescent fathers 


□ 


An alternative public school serving 


□ 


Peer support programs 




pregnant/parenting adolescents exclusively 


□ 


Grandparents* programs 


□ 


An independent school (e.g., religious) 


□ 


Mentoring 


□ 


A church or other religious organization 


□ 


Other 


□ 


A hospital or other health facility 






□ 


A city, county, state or other government agency 


4. 


The geoographic area served by the program 




(e.g., County Department of Health) 






□ 


A community-based non-residential facility (e.g.. 


□ 


Inner city 




community center or agency providing social 


□ 


Urban 




services) 


□ 


Suburban (Town/City) 


□ 


A community -based residential facility 


□ 


Rural 


□ 


Home visiting 






□ 


Other 


5. 


The populations predominantly served by this 








program 


2. 


Who the program serves 


□ 


African American 






□ 


Asian American 


□ 


Pregnant adolescents 


□ 


Hispanic American 


□ 


Adolescent mothers 


□ 


Native American 


□ 


Adolescent fathers 


□ 


White Non-Hispanic 


□ 


Infants bom to adolescents 


□ 


Other 


□ 


Toddlers bom to adolescents 






□ 


Preschoolers bom to adolescents 


6. 


The age range served by the program 


□ 


Grandparents and other relatives (families of 








adolescents) 


□ 


Elementary school (Younger than 1 1 years) 






□ 


Middle school (11-13 years) 


3. 


The services offered by the program 


□ 


Secondary school (14-18 years and older) 




Health Care: 


7. 


The length of time the program has been in 


□ 


Prenatal care 




existence 


□ 


Well-baby care 






□ 


Health care for adolescent parents, their children. 


□ 


Less than 3 years 




and their families 


□ 


3-5 years 


□ 


Health education 


□ 


6-10 years 


□ 


HIV/AIDS education 


□ 


1 1-15 years 






□ 


More than 1 5 years 




Child Care/Early Childhood Education: 






□ 


Parenting classes for adolescent mothers 


8. 


Whether the program can offer documentation for 


□ 


Parenting classes for adolescent fathers 




its success in at least one site, such as evidence 


□ 


On-site Infant, toddler and/or preschool childcare 




that it: 


□ 


Early childhood education by certified teachers 










Yes 


No Don’t Know 




Educational/Vocational Programs: 


□ 


□ □ Decreases the school dropout 


□ 


Academic educational programs for adolescent 




rate of adolescent parents 




mothers and/or adolescent fathers 


□ 


□ □ Improves the health and well- 


□ 


Vocational education/training 




being of children bom to 


□ 


Job placement 




adolescent parents 


□ 


An academic diploma 


□ 


□ □ Decreases repeat pregnancies 


□ 


Career counseling 




of adolescent parents 






□ 


□ □ Increases the economic self- 








sufficiency of adolescent 








parents 
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9. 


Is this program now, or has it ever been, 
supported at least in part by federal or foundation 
sources of funds? 


10. 


Has this program been replicated in whole or part 
by other schools or communities? 








Yes No Don’t Know 




Yes No Don’t Know 

□ □ □ 




□ □ □ 




IT COPY AVAILABLE 
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Group I: New Futures School 



Name 


New Futures School 


State 


Albuquerque, NM 


Location 


Urban, suburban, rural 


School Based 


Alternative public school 


Racial/ethnic populations served 


African American, Hispanic American, 
Native American, White Non Hispanic 


Prenatal care 


Yes 


Health care 


Yes 


Child care 


Yes 


Early childhood education 


Yes 


Academic/vocational programs 


Yes 


Case management and other social services 


Yes 


Documentation 


All criteria 


Replicated in other sites 


Yes 


Numbers served 




Age of program 


15+ years 


Contact person 


Dr. Sandy Dixon, Principal 



Program Objectives. New Futures School. Albuquerque, NM.. It serves elementary, 
middle and secondary) school pregnant and parenting teens, their children and extended 
families in an alternative school setting. 

Program Components. Offers parenting classes for adolescent mothers, some social services, 
counseling, peer support programs and mentoring. Has been supported. Has been replicated. 





Group I: Paquin Family Health Center 



Name 


Paquin Family Health Center 


State 


Baltimore, MD 


Location 


Inner city 


School Based 


Alternative public school 


Racial/ethnic populations served 


African American, Non White Hispanic, 
White 


Prenatal care 


Yes 


Health care 


Yes 


Child care 


Yes 


Early childhood education 


Yes 


Academic/vocational programs 


Yes 


Case management and other social services 


Yes 


Documentation 


Three indicators 


Replicated in other sites 


Possibly 


Numbers served 


800 mothers 


Age of program 


3-5 years 


Contact person 


Dr. Beth Bamet 



Program Objectives. The Paquin Family Health Center lodged in an alternative school 
setting provides services to all populations listed in survey. 

Program Components. Full service program. Can provide data that it improves health and 
well-being of children bom to adolescents, decreases repeat pregnancies. Does not track those 
who leave the program. 
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Group I: Ohio Department of Education Grads 
(Graduation, Reality and Dual Role Skills) 



Name 


GRADS: Ohio Department of Education 


State 


Ohio 


Location 


All areas - national program 


School Based 


Public schools 


Racial/ethnic populations served 


All populations 


Prenatal care 


Yes (some programs) 


Health care 


Yes (some programs) 


Child care 


Yes 


Early childhood education 


Yes 


Academic/vocational programs 


Yes 


Case management and other social services 


Yes 


Documentation 


All four indicators 


Replicated in other sites 


Yes 


Numbers served 




Age of program 




Contact person 


Sharon G. Enright 



Program Objectives. The GRADS (Graduation, Reality and Dual Role Skills) Program, 
originally developed by the Ohio Department of Education, serves pregnant and parenting 
middle and secondary school age adolescents. A prevention component is offered in middle 
schools. 

Program Components. The GRADS Program is offered by GRADS trained teachers in 75% of 
the public schools in Ohio. Some of the programs include prenatal care. The original GRADS 
Program has been recognized as exemplary by the National Diffusion Network (NDN). 



ERIC 
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Group I: Grads: New Mexico 



Name 


GRADS: New Mexico 


State 


New Mexico 


Location 


Mostly rural 


School Based 


23 public schools, including alternative 
schools 


Racial/ethnic populations served 


All populations, including several sites 
serving Indian reservations. 


Prenatal care 




Health care 


Yes 


Child care 


Yes 


Early childhood education 




Academic/vocational programs 


Yes 


Case management and other social services 


Yes 


Docmnentation 


All four indicators 


Replicated in other sites 


Yes 


Numbers served 




Age of program 


6-10 years 


Contact person 


Sharon Waggoner 



Program Objectives. The GRADS Program in New Mexico serves pregnant and parenting 
adolescents and their infants, toddlers, preschoolers, and extended families. The program also 
seeks to prevent adolescent childbearing. 

Program Components. The GRADS Program is lodged in 23 public schools including 7 
alternative schools. NM is mostly rural. Several sites serve Indian reservations. The pregnancy 
prevention component is middle school. In existence since 1 989. Included with materials is the 
Taking Charge News. 





Group I: Margaret Hudson Program, Inc 



Name 


Margaret Hudson Program, Inc. 


State 


Tulsa, OK 


Location 


Suburban, rural 


School Based 




Racial/ethnic populations served 


African American, Native American, White 
Non Hispanic 


Prenatal care 


Yes 


Health care 


Yes 


Child care 


Yes 


Early childhood education 


Yes 


Academic/vocational programs 


Yes 


Case management and other social services 


Yes 


Documentation 


All four indicators 


Replicated in other sites 


Yes -multiple sites 


Numbers served 




Age of program 




Contact person 


Jan L. Figart, Executive Director 



Program Objectives. Margaret Hudson Program, Inc. Tulsa, OK is a private non profit 
conducting alternative education, health and social services for pregnant, parenting teens, their 
children and extended families. Middle and secondary school, ages 9-21. 

Program Components. Coordinated well baby care, health and HIV/AIDS education. 
Parenting classes. Multiple facilities including 3 alternative school programs. 
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FIGURE 3: LOCATION OF PROGRAM 
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FIGURE 4: SERVICES OFFERED 
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FIGURE 5: DOCUMENTATION 
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FIGURE 6: AGE OF PROGRAMS 
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FIGURE 7: REPLICATION 
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Reactions to the Expert Panel System as Piloted by OERI 



Lynn H. Fox 
American University 



The Office of Educational Research and Improvement at the United Sates Department of Education 
is in the process of creating a system to aid the educational community in its search for educational 
excellence and change. This has begun with a pilot test of two “Expert Panels” to review 
submissions from the field of programs, practices, and materials. Submissions are to be evaluated 
on criteria described elsewhere and to be considered for the designation of either “exemplary” or 
“promising” as described elsewhere. As a member of the pilot “Expert Panel for Gender Equity” and 
chair of its subpanel on mathematics, science, and technology I have written this paper to share my 
views on this effort to date. The paper will deal with the following three questions: 

• What are some of the issues involved in the expert panel evaluation process that need to be 
addressed? 

• What are some ways to disseminate information about programs, products and practices to 
the educational community? and 

• How would I organize the review and dissemination process to address some of the issues? 

What are some of the issues involved in expert panel evaluation process? 

I believe a question that keeps coming up is, “Why should program developers want to submit their 
programs for review?” As the pilot effort in gender equity in education has proceeded I have been 
struck with the realization that the submission process can involve a great deal of work for the 
submitter. Some potential submitters have developed a program or materials as part of a grant and 
may not have funds available for further development, evaluation, or diss eminati on Few programs 
have the potential to generate significant income for persons or institutions. Therefore any long term 
benefits of extensive evaluation may not seem worth the cost. I applaud the suggestion by Lois Ellen 
Datta shared on the Gender Panel LISTSERV that programs be given $1000 toward the cost of 
conducting evaluations. 

A minor but not unimportant issue involves the communication between panels and submitters. We 
need to set up some procedure, perhaps mirroring what is done with submissions for publication to 
scholarly journals. The receipt of submissions would be formally acknowledged and a time frame 
for review should be stipulated. Requests for additional documentation should be channeled from 
reviewers to panel chairs to a central support staff person who in turn would request the additional 
information. In fact the central support staff should screen submissions for completeness in the 
beginning and try to improve the quality of the submission before it is externally reviewed. 



A much more serious issue is “Who are the appropriate persons to review my submission and how 
can conflict of interest be avoided?” In the beginning we are dealing with many people who know 
each other and the field. As this effort expands it would be wise to institutionalize a process that 
monitors the assignment of materials to reviewers. A blind review process is not really possible in 
many instances. For example, it is not possible to conceal the authorship of a book. Many programs 
have been around long enough that descriptions of them without the names of the key individuals 
would not mask their identity. We need a form for reviewers to sign stating that they have no vested 
interest in the rating of the submission (i.e. they have not been involved with the project in any way 
or are not involved with a competing project or product.) Although I believe the names of the 
specific reviewers of a particular submission should not be public, a list of reviewers and panel 
members should be. If I were to submit a program, I would want to know who my “peer judges” 
were to be and feel free to request that someone not be a reviewer of my project if I feel they may 
have a conflict of interest. 

Also, we need to think about whether or not reviewers should be paid? I believe that they should 
be paid. I think the process is more time consuming than reviewing a journal article. Unless they 
are paid, OERI will not be able to hold them to tight time schedules for turn around of evaluations. 

The issue which has been most often discussed by participants in the pilot effort for the Gender 
Equity Panel is the distinction between promising and exemplary. Is it a clear distinction? Is it 
reasonable? I believe the distinction will work for the submissions of programs but is problematic 
when we deal with the categories of practice and products. First, practices are unlikely to be on the 
scale of programs. Many things, especially those related to teacher education, are not likely to have 
extensive documentation about effectiveness. Many things that might fit in this category are 
simulations and activities for use in pre-service or in-service training programs. I believe that the 
best way to collect, evaluate and disseminate them may be to advertise for these through meetings 
and publications of professional organizations such as NCTM, ATE, AERA, etc. for inclusion in a 
published volume called something like; Notes from the field: Lessons for equitable teaching . 
Perhaps one of the organizations would take on the publication of the first one. I am envisioning 
something similar to two volumes of teaching ideas for psychology that were compiled like this and 
published by the American Psychological Association. The expert panel could review the 
submissions and decide which ones to include but not apply the same rules of evidence that are being 
used for programs. 

An example of what might be submitted in the category of practices is a simulation for a 
mathematics education class that I saw presented at the meeting of the Association of Mathematics 
Teacher Educators ( AMTE) in February of this year by two college professors. Although they have 
informally evaluated this simulation by looking at samples of students work and written reactions 
by students to the activity for one semester, they clearly have not amassed enough data to be 
included as “exemplary” under the current guidelines. Nor are they ever likely to have that type of 
data. This simulation portrays a teacher who is unaware of her sexist teaching behaviors as she sends 
negative messages to girls while paying more attention to boys. Pre-service teachers must take the 
roles of students (without a script) and experience the “discrimination” first hand. Much of the value 
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of this exercise would depend upon the individual instructor’s skill in conducting the simulation and 
their ability to handle the post demonstration “debriefing.” The activity could be evaluated by the 
Expert Panel for its appropriateness, interest level, and accuracy in the portrayal of an issue in gender 
equity but it can never be fully evaluated in terms of its potential to change attitudes and behaviors 
of pre-service teachers when implemented by someone else. 

If activities like the one described above could be called “promising” based on their attempt to deal 
with a problem well documented in research (that of differential treatment of girls by teachers), they 
could be published on a web page or disseminated on a LISTSERV to other teacher educators so that 
feedback about their use by others could be collected. After several rounds of this, a volume of 
“most successful” activities could be compiled. From informal conversations I have had with 
college professors who train secondary mathematics teachers or elementary school teachers, I feel 
this would be well-received. This approach might help encourage more submissions of practice as 
well since the possibility of having the submission published would be a real incentive. 

Products ( books, kits, software, audio tapes, or video tapes) pose their own challenges. Because 
of the cost involved in producing these materials, they are likely to be sold commercially. I feel we 
should be very careful in how the federal government is used to promote products prepared by a 
mixture of not-for-profit and for-profit organizations. Also I don’t think that such materials can be 
evaluated with the same set of criteria we are trying to use for programs. Textbooks for example 
would not normally be expected to amass the kinds of formal evidence we are requiring. I think one 
way of handling this might be to have a more public evaluation of these materials than is planned 
for the other two categories. Currently such materials are likely to be reviewed in publications of 
professional organizations. The task of the Expert Panel might be to compile all the reviews of 
materials like books and reference those in the panel’s own review of the material in the ERIC 
database. I think the distinction of promising versus exemplary should be dropped for this category 
and another term substituted such as “quality resources.” 

I think an issue we have not addressed is “timeliness” of evaluation and evolving status of programs, 
practices or materials over time. What does the receipt of the designation of either promising or 
exemplary in a given year mean in subsequent years. For example, the success of a program might 
be dependent upon the commitment, vision, or energy of one specific “key” person. A program 
that is rated as exemplary in 1997 could conceivably be radically changed by 1998 due to changes 
in that “key” person’s role in the program. Another example would be a textbook written in 1 997 
that was judged to be terrific but becomes “out-of date” within a few years. On the other hand it 
would not seem practical to re-evaluate every program, practice, or product every year to “re-certify” 
status. It has been suggested that programs, products, and practices that are rated as “promising” 
would seek support to conduct the replications and evaluation necessary to reach “exemplary” status. 
This may or may not prove to be the case for some or all submissions. I think the panel needs to 
discuss this issue and come up with a policy and state that policy in both the materials that are used 
to request submissions and in materials used later for dissemination of evaluation results. For 
example, we might list “promising” programs or practices for a three year period after submission, 
and then require re-submission for inclusion. The time period for exemplary programs, etc. might 
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be five years. Programs that experience any significant changes in personnel or program 
components should be required to notify the panel to that effect so that any listings of these programs 
are in some way annotated to announce these changes since the awarding of their designated status. 



My final concern is that we better prepare the panel members and reviewers for the process. I think 
we need to do this by adding two pieces to the process. First, either the panel or OERI should 
commission a review of the literature on the “problem” to be addressed before we call for 
submissions. This might be coupled with some surveying of parents, teachers, and others as to their 
perception of critical questions or problems. For example, in the area of gender equity we should 
have started with a review of the literature to help focus the nature and extent of the “problems” and 
the types of outcome variables that are most critical. This would allow us to think more in terms of 
creating bench marks or quality indicators. Then when we call for submissions we might be more 
specific in a given year as to what types of outcomes exemplary projects should address. Once the 
paper was written the panel members could meet to discuss the research review. Second, we need 
some training on how to review. While I would prefer a face-to-face process, it is possible to do it 
using some combination of telephone conferencing, e-mail, and perhaps online chat rooms. These 
additions to the process would give more focus to the process. 

What are some ways to disseminate information? 

Announcements or reports that are developed to inform consumers about programs, practices, and 
materials must be done in forms and formats that can be easily and frequently updated. I believe 
there are several important factors we need to consider. These include funding, involvement of the 
educational community as a whole, and the need for different approaches for the different categories 
of submissions. 

For programs, I think a database with limited information about them should be created by OERI 
and accessible in some manner through ERIC on the INTERNET. I propose that ERIC create a hew 
category other than EJ ox ED to be EP (for Educational Programs). This data base could be updated 
on a monthly basis. This would allow for a rolling submission process and the aimotating and/or 
expunging of records in the data basis as needed. The consumer reports that detail the evaluation 
of a EP record could be put on the system just like ED documents and also made available in paper 
formats. I suggest that the paper formats be done somewhat like a serial or newspaper publication 
so that one could put out print information on new EPs as they emerged every three months or six 
months and continue to list (but not review) all other designated EPs for that time-frame. So if I was 
a school system looking for a good program to interest more high school girls in careers in science, 
math, and technology, I could go on to ERIC and search for programs in that category and see the 
accompanying review and information as to whom to contact if I want to purchase materials or 
services. 

I think we need to be creative in our approach to recognizing and sharing information about practices 
and products. In my previous discussion of the need for different evaluation approaches for 
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practices, I suggested a different format for dissemination using the publication of a book of 
“promising practices” on an annual basis. I am uncertain as to what would be the most effective 
approach for the category of products. Perhaps several different approaches should be tried and 
compared. 

Another important issues is funding. Sufficient funds are are needed to pay for the development of 
a variety of dissemination activities. National attention needs to be focused on the Expert Panel 
effort and outcomes. It takes time for educators to learn about new sources of information. I believe 
OERI needs to launch an ambitious publicity effort to draw national attention to the availability of 
the dissemination process(whenever it all gets sorted out) and have ceremonies and press 
conferences to announce the results of each years’ search for exemplary and promising practices in 
education. I believe that for all three types of submissions (programs, practices, and products) and 
for all of the stages of the process (calls for submissions, reviews of submissions, and 
dissemination) the educational community as a whole must be involved through the existing 
networks of professional organizations, colleges and universities, and state and local agencies. I feel 
we have attempted to do this informally on a very tiny scale in the pilot for gender equity. Our 
efforts have been well-received so far but need to be systematized and greatly expanded once the 
panels become fully operational. For example, I made a presentation about the Expert Panel work 
in the math, science, and technology subpanel of the Gender Equity Panel at the February, 1997 
meeting of the Association of Mathematics Teacher Educators in Washington, D.C. They have 
requested that we give them an article for their organization’s newsletter and they have a web page 
that they would be delighted to link to the Expert Panel when (and if?) it is created. 

Indeed I believe the creation and maintenance of an EXPERT PANEL website is critical (perhaps 
through WEEA as the current submission information is being done now). This site might link to 
the data base described above as well as be a link from every website for every professional 
educational organization such as the National Council of Teachers of Mathematics, the Association 
of Math Teacher Educators, etc. This would also serve as a tool for recmiting submissions. 

How would I organize the review and dissemination process? 

Review Process 

• Identify one or two key issues or concerns each year for each panel. A survey of the 
profession and the public could help focus the questions and priorities. 

• Commission a review of the literature on the issue(s). If necessary there could be a review 
of the literature each year for each panel. 

• Convene each Expert Panel for a working conference each year at which time they discuss 
the literature review and practice reviewing submissions. 

• Conduct work of the panel by mail, e-mail, LISTSERV, telephone, and conference calls. 
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• Convene a meeting of chairs of the panel and subpanels, with OERI staff to discuss the 
evaluations and ratings of the submissions prior to dissemination activities. This should be 
done panel by panel. (Expert panel members might be chosen for two or three year terms 
in a staggered pattern to maintain overlap each year.) 

Alternative Review plan: 

Follow all the steps above but combine III, IV, and V in one week long wor kin g conference. 

Dissemination: 

I. Create a category within the existing ERIC system to provide an online database of promising 
and exemplary efforts. This would contain a brief abstract and contact information. A longer 
“consumer report” document be made available from GPO and updated every year. 

II. Create other dissemination materials such as web page and brochure announcements for 
programs and products. This might be done in conjunction with various professional 
organizations. For example, a web page might list all projects for a given panel that had been 
designated promising or exemplary with e-mail addresses for them, but each month the page 
could provide more in-depth information on one specific program or category of programs. 

III. Create an annual publication of “promising practices” for each panel. 

IV. Create a series of awards and events to publicize the system of Expert Panels and the particular 
recognized submissions in a given year. This might include Public Service Announcements for 
Radio and Television, cash awards to organizations, and other types of recognition. For 
example, OERI might present awards at major professional organization meetings such as 
National Council of Teachers of Mathematics, the American Educational Research Association, 
the Association of Teacher Educators, etc. 

Conclusions 

Although there are some unresolved issues with the system of Expert Panels being piloted by OERI, 

there is much to recommend it. In order for it to be effective I believe the following four things need 

to be done: 

• Streamline the submissions and review processes. To do this I have recommended several 
things. The most important is to apply the “promising” versus “exemplary” distinction to 
programs only. This will mean revising the criteria and processes for evaluation for 
submissions of practices and products. 

• Tie dissemination to existing mechanisms and organizations as much as possible. I 
recommend that the dissemination of information for all three categories utilize the ERIC 
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system to some degree. Use the INTERNET to link information to other organizations that 
maintain web pages. Use a variety of “print” formats in a cost-conscious manner. I also 
believe that the information that would be provided in a database will be different for each 
of the three categories. In addition, for practices, I recommend the development of an 
annual collection of promising practices published perhaps in conjimction with one or more 
professional organizations. 

• Maintain a close working relationship between OERI and the educational community in the 
selection of members of the panels and in the oversight of the work of the panels for both 
review and dissemination. This can be done in a number of ways and, at the very least, 
should include having professional organizations nominate a member of their association 
to serve on one or more panels. Unless state and local educational agencies and professional 
educational organizations become involved in the process at every level, the system is not 
likely to be widely known and used. 

• Formalize fiscal and policy issues and communicate this to the educational community at 
large. These issues include selection of panel members, payment for reviews of submissions, 
timeliness of a promising or exemplary classification, conflict of interest, and funding for 
submissions to defray the costs of evaluation and submission, especially for those who seek 
to move from the status of “promising” to “exemplary.” 

Efforts to help educators in schools and school systems locate information about programs, practices 
and products that are of high quality are laudable. It is important to realize that there may be some 
problems in making judgements about the “effectiveness” of particular programs, practices, or 
products for a variety of reasons. While only longitudinal follow up of students who participate in 
a program as compared with similar students that did not participate may be the type of evidence we 
would like to see available, the reality is that many excellent programs have not had the resources 
to conduct such investigations. The desire for technical accuracy in the claims a program or product 
makes is critical, yet it must be balanced against some standard of “reasonableness” of effort. Much 
of the work in the field has not been done in a vacuum but has emerged out of years of serious 
research. Thus this effort to develop consumer reports must be guided by that collective body of 
research. 

The success of this effort must ultimately be measured in terms of its use by the public it seeks to 
serve. Thus, the emphasis on quality reviews must be balanced with concerns for the methods and 
quality of dissemination efforts. Reports must also be readily accessible and easy to imderstand. 
Dissemination efforts should focus on creating meaningful links with the educational community 
through professional organizations and educational agencies at every level. 



Some Thoughts and Reflections an the OERI Expert Panel System 



Harilyn Rousso, Chair, Gender Equity and Disability 
Subpanel of the Gender Equity Panel 
May 1997 



Paper in Response to the OERI Expert Panel Meeting at AERA 

• Brainstorm ideas on the potential of an overall Expert Panel on Special Education and 

Disability in the Department of Education, with subpanels on important issues like gender 
equity. 

I am an activist, advocate, trainer and program developer who has worked in and with a 
variety of nonprofit agencies outside of government, mainly on issues of disability equity, 
gender equity, and their intersection. I currently own my own consulting business. Disabilities 
Unlimited Consulting Services, providing training, public education, action-oriented research 
and advocacy. My contact with the U.S. Department of Education has been fairly limited, 
involving the roles of fund seeker, grantee/subcontractor, peer reviewer, trainer, and 
resource/information seeker. While my present role as Chair of the Gender Equity and Disability 
Subpanel, has given me somewhat more contact with the Department of Education in general, and 
OSERS in particular, since OSERS is providing supplemental support for the Subpanel, I speak 
largely as an outsider, as a consumer and producer of resources in the field of disability, and 
specifically disability and gender. 

As a consumer, looking for materials for my teaching, training and self-education, and as a 
producer, looking for diverse and creative dissemination strategies, I have felt somewhat imclear 
about how to most effectively use the Department of Education as a resource. There appear to be 
many individuals, programs and aspects of the Department working on and funding 
disability-related programs, practices, policies and materials, some in unique and some in 
overlapping ways. They use a range of criteria to evaluate quality and effectiveness of programs 
and resources; reflect a range of orientations toward disability, including 
deficiency/health/medical and sociopolitical/civil rights models, reflecting changing attitudes and 
perspectives over time; and have evolved a range of formal and informal dissemination systems 
for programs and resources developed and/or funded by the Department, with no central 
clearinghouse, at least none that is readily obvious to an outsider. In the face of such complexity, 
the question becomes how to begin to tap the richness of the work of the Department. 

An Expert Panel on Special Education and Disability could offer consumers and producers the 
ideal place to begin. Among other things, it could bring together the diverse disability-related 
work in education involving program and materials development going on in the Department to 
a central forum, where panel members could develop and apply broad criteria for effectiveness. 



qiiality, significance and usefulness, setting standards for opromisingo and oexemplaryo resources 
that consumers could readily recognize and use in their selection process, and producers could 
incorporate into product development. Because the panel membership would represent diversity 
on many dimensions, including professional/personal expertise, disability status, gender, 
race/ethnicity, class, encompassing those from outside of and within the Department, and would 
be complemented by a diverse group of advisors, the criteria and review process would reflect 
the voices of many, including those not often heard, for example students with disabilities. In 
addition to developing uniform criteria and a common understanding of opromisingo and 
©exemplary ,6 the Panel would develop a uniform and centralized dissemination system to spread 
the word about promising and exemplary materials in the field of disability and education. Thus, 
consumers would readily know how to find the high quality resources they seek, and producers 
of quality resources would be assured of a comprehensive dissemination strategy. 

The Panel, in addition to reviewing resources developed and/or funded by the Department, would 
also identify and review relevant materials developed outside the Department, offering several 
advantages. For consumers, the Expert Panel would then truly serve as a clearinghouse for quality 
resources, regardless of the resource funding source; and for independent product developers who 
often lack funds for dissemination, the seal of opromisingo or oexemplaryo would entitle products 
to no-cost (or low-cost) dissemination through a centralized system. In addition, independent 
resources identified and reviewed by the Panel could serve as a catalyst to the Department, 
uncovering up-and-coming issues and trends, and programs and products worthy of government 
support. 

Given the diversity of the work in disability and education both within and outside of the 
Department of Education, it would make sense for the Expert Panel to encompass a series of topic 
or issue-specific Subpanels, consisting of select members from the Panel and Advisory Group 
with appropriate expertise. Such Subpanels might then take the broad criteria for oexemplaryo 
and opromisingo set by the Panel, and make them specific to the topic/issue they are addressing. 
Ideally such Subpanels would be in close communication with one another, learning from one 
another and ensuring consistency across topics and issues. Also, such Subpanels might serve as 
stimuli to the Department, disability and education field and public at large, introducing new 
and/or underaddressed topics. For example, the Gender Equity and Disability Subpanel of the 
Gender Equity Panel has as one of its goals to raise consciousness in the gender equity community 
that gender bias is an issue for students with disabilities. A similar Subpanel might be developed 
in the Special Education and Disability Panel, serving to raise consciousness in the disability 
community, who often fail to recognize that students with disabilities have gender, and that 
gender affects access, treatment and outcome in disability services. 

How might criteria of all expert panels best attend to equity based on disability? 

While it would be useful to have an Expert Panel on Special Education and Disability, it is equally 
useful and essential that issues of disability equity be incorporated into the work of all Expert 
Panels so that the historic pattern of segregating and isolating people with disabilities and their 
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needs and concerns is not perpetuated. All submissions for all panels, regardless of whether they 
have been specifically designed with people with disabilities in mind, should be assessed for 
whether they promote disability equity, whether they avoid perpetuating disability bias, and 
whether they are appropriate for an inclusive learning environment that includes learners with and 
without disabilities, reflecting current educational trends and laws. The review criteria for all 
panels should include specific reference to disability equity issues under one or more of the four 
criteria categories currently in use (effectiveness; quality; educational significance; usefulness). 
Specificity is necessary because disability is still a relatively new equity category that is often 
overlooked by those not working in the disability equity field. Those unfamiliar with disability 
may fail to recognize that terms like all students, a wide spectrum of students or underserved 
studentsd includes students with disabilities, and that all forms of bias, discrimination or 
stereotypingd includes disability bias, discrimination or stereot)q)ing. Where relevant, issues of 
accessibility of materials to users with disabilities, e.g. alternative formats for written materials, 
such as on disk, audio tape, large print and Braille, or disability-related accommodations for 
hands-on activities, should be incorporated into review criteria, as should evidence of co mmitm ent 
to the spirit and requirements of disability-rights laws, such as IDEA and the ADA. Thus, for 
example, the review criteria for program replication guides should include consideration of 
whether the authors or developers encourage the selection of an accessible program site, the 
provision of accessible transportation to participants, and the provision of sign language 
interpreters. 

To help ensure that all Expert Panels address disability equity issues, it is important to include 
people who are experts in this area, ideally people who are themselves disabled, among the 
official members and/or advisors of each Panel. Related to this, disability status and expertise 
in disability equity should be recognized as essential dimensions of diversity. 

Ideally, there would be an interdependent relationship between the Special Education and 
Disability Expert Panel and the other Panels. Members and advisors to the Special Education and 
Disability Panel could offer their expertise to other Panels in terms of suggesting potential 
members knowledgeable in disability equity and advising on the crafting of review criteria so that 
they appropriately reflect disability equity issues. The members and advisors of other Panels 
could in turn offer their expertise in particular content areas. In addition, some submissions to the 
Special Education and Disability Panel would be referred to other Panels reflecting the content 
area of the submission, so that there could be joint reviews. Thus, for example, a submission 
involving a math and science program for special education students would be reviewed by the 
Special Education/Disability Panel and the Mathematics and Science Education Expert Panel. 
Not only would such a joint review process be likely to enhance the quality on the review, but it 
would also reinforce the notion that promoting quality education for students with disabilities is 
the responsibility of all educators and all those committed to quality education for all. 

Is there a need for a superstructure to manage all current and future panels, and if so, 
what should its responsibilities be? 
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I believe that a superstructure that assumes a facilitative and coordinating role could be extremely 
helpful to the growth, development and success of the panel, system. Tasks might include 
identifying key and/or cutting edge topics or issues in education in which new Expert Panels 
should be developed; developing an initial pool of potential members and/or advisors for new 
panels; serving as a resource for new and ongoing Panel Chairs on diverse aspects of Panel work; 
serving an oversight function in ensuring the review criteria and the definitions of opromisingo 
and oexemplaryo involve a basic level of consistency across Panels; facilitating comm un ication 
and cross-pollination across Panels; and serving as an advocate and publicist for the Panels and 
their work, including assistance in obtaining supplementary funds to cover their time and expenses 
so that they can justify their absence from other, often paid, responsibilities. 

It has been my sense as a Subpanel chair that Panels and Subpanels take their work extremely 
seriously, set high standards for themselves (often higher than any external body could impose), 
are committed to meeting these standards in a timely, cost-effective maimer, and are not in need 
of any outside regulation of the quality and quantity of their work. A high degree of autonomy 
seems to be an essential ingredient in the effective functioning of the Panels. What Panels and 
Subpanels are likely to need most from a superstructure is support for the important, intriguing 
and often difficult tasks and challenges that they face. 

Ideally, the superstructure would be made up of diverse representatives from the various 
departments/programs of Department of Education and the various Expert Panels so that there 
would be a broad range of expertise upon which to draw. 

How can we make the submission requirements and process better? 

The good news and the bad news about the Submission Guidelines is that they are comprehensive, 
extensive and exhaustive. As a result, they are highly labor-intensive and time consuming to 
complete. Many submitters and potential submitters are overextended in their work schedules and 
have little time or resources to devote to additional undertakings. The fact that the Expert Panels 
are new, so that they are not well known or understood, and their positive potential not readily 
recognized, further discourages the use of already limited resources for the task of developing a 
submission. There may also be a negative incentive discouraging submissions, the fear that if a 
submission is not designated as promising or exemplary, the developer may be deprived of future 
funding. 

Some solutions might include the following. First, continue to streamline and simplify the 
guidelines so that less extensive information is required and that which is required is easier to 
vmderstand and provide. The currently suggested length of 10-15 pages is likely to be a 
discouragement; 5-10 pages may be more reasonable. To the extent possible, there should be 
one question rather than multiple questions under each section of each criterion, with several 
examples to clarify the intent of the question, and perhaps a yes/no checklist to get at certain types 

of information (e.g. the materials are accessible to people with disabilities: ^Yes No) 

Including a mock submission or some mock answers to certain questions might also be useful. 
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Next, continue to publicize the goals, purposes and positive contributions of the Expert Panels, 
particularly among non-research resource developers working in the field. The publicity needs 
to work on demystifying the notions of evaluation and measures of effectiveness, to break down 
the historically antagonistic relationship between evaluators and program people, and to help 
program and resource developers realize that to some extent they already are evaluating and 
considering the effectiveness of their own work, as they seek to make it better, and that the Panel 
process offers them the potential to engage in such reflection and assessment more systematically. 
The publicity (and probably also the submission gtiidelines) needs to clearly describe the 
advantages of submitting and of receiving promising or exemplary designations, cover, for 
example, such issues as how information about resources designated as promising or exemplary 
will be disseminated; and how such designations might affect sales and/or fuhore funding 
opportunities. It might be particularly useful to develop a mechanism whereby resources 
designated as promising would have ready access to replication/evaluation funds or at least have 
an advantage in grant competitions. Along related lines, it might be helpful to incorporate 
information about the Expert Panels in upcoming RFAs, and include questions on the application 
about how the applicant, if funded, might work with the Expert Panel system, and (later on in the 
Expert Panel process) whether any of the applicants products have received designations of 
promising or exemplary. 

To help spread the word, all Panel members and advisors could be engaged in the marketing 
process, bringing information to all their other professional activities. A brief flier or press 
release, in readily understandable terms, that Panel members and advisors could distribute and/or 
adapt to their particular Panel or Subpanel might be useful. 

Finally, in terms of the disincentive issue, there needs to be a commitment within the Expert Panel 
system to keep the list of submissions confidential, and to develop a mechanism to provide 
detailed feedback and technical assistance to the developers of those submissions that do not 
achieve the promising or exemplary designations, if the developers want it. The publicity 
information should indicate that unlike grant and other competitions, there is no limit to the 
number or percentage of materials designated as promising or exemplary, and that intent is to 
designate as many resources as meet the criteria. Hence, there need not be winners and losers, but 
rather winners and those with potential to win if they are willing to engage in additional work. 

Hopefully, over time, as the loser-winner image for the Panels is transformed into a win-win 
model, there may be no need to keep submissions confidential. 

How can we identify standards of effectiveness in the area of gender equity in general and 

gender equity and disability in particular, given that no national standards exist? 

The extensive body of research documenting the extent, nature and impact of gender bias in 
education, which has focused primarily on nondisabled students, and the small but hopefully 



growing number of studies on gender bias in special education and other educational settings 
serving students with disabilities, provide a good place to begin in terms of crafting standards of 
effectiveness for gender equity resources in general and gender equity and disability resources in 
particular. Thus, effective resources would include those that demonstrably reduce or eliminate 
one or more types of gender bias in access to, treatment in and outcome of educational 
experiences, as described in the literature. 

One needs to be somewhat wary about applying the results of research done with nondisabled 
students to the experiences of students with disabilities, both because until relatively recently, the 
special education system has been quite separate fi-om general education, with its own rules, 
regulation, style and climate, which may influence the nature and degree of bias, and because the 
interaction of disability bias and gender bias adds an additional level of complexity not reflected 
in the generic research. These factors make a compelling argument for fostering more research 
on gender equity and disability. Ideally, such research would involve researchers with disabilities, 
who could offer an important perspective not currently reflected in most gender equity research. 
Because further research is a longer term goal, in the interim, landmark theoretical work on gender 
equity and disability, for example work by Adrienne Asch and Michelle Fine, and by Jenny 
Morris, as well as anecdotal information from activists working in the field, might prove helpful 
in developing indicators of effectiveness. Seminal theoretical work in the gender equity field 
might be similarly useful for developing the generic gender equity criteria. 

In addition, for the Gender Equity Panel in general and Gender Equity and Disability Subpanel 
in particular, submissions themselves could be an important source of indicators of effectiveness. 
Resource developers may have their own vision of what constitutes effectiveness and may have 
designed their resource with that vision in mind; also, unintended beneficial effects of their 
product may extend the understanding of effectiveness and expand their vision still further. 
Whether or not they have such a vision, the way that submitters answer the questions on the 
Submission Guidelines about effectiveness, the measures that they offer, may spark the thinking 
of and expand the definition used by reviewers. It is useful to think about the measures of 
effectiveness for gender equity as a work in progress in which new research, new theory, new 
submissions all help to grow the list. 



• How should we use Expert Panels to review seminal research syntheses, videos, etc. that 

convey key information about the field? 

While seminal research syntheses, studies, reports, videos or other materials that provide a 
snapshot of a field that is the focus of an Expert Panel or Subpanel are not likely to qualify as 
formal submissions because they lack indicators of effectiveness, they may serve important 
functions to the work of the Expert Panel system. At the very least, it should be standard practice 
for all Panel/Subpanel members to read such conveyors of basic information to ensure a certain 
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level of commonality and equity of understanding and to facilitate the discussion of measures of 
effectiveness, particularly when there are no standards of effectiveness. To save time and energy, 
it would be important to limit the number and length of those documents to be read by all. The 
Chair might ask for recommendations from Panel/Subpanel members, and then a collective 
decision reached about which to pursue. In addition, a list of the information resources used by 
each Panel/Subpanel, with a brief summary for each item, should be made available for new 
Panel/Subpanel members, as part of the orientation, to members of other Panels/Subpanels, as an 
introduction to the field, and to consumers, who might wish to incorporate inf ormation from some 
of these materials into resources they are developing. 

One potential difficulty with sharing such lists with consumers is that the appearance of an item 
on the list becomes an endorsement, whereas the materials have been used but not formally 
reviewed and critiqued by the Panel/Subpanel. This may set up a double standard, in which 
resources that are submitted must be formally reviewed and designated as pro mis ing or exemplary 
before being disseminated, whereas informational materials do not have to undergo such a 
process. One compromise might be for the informational resources to undergo a review, but one 
that is less rigorous and time consuming; for example, each item on the information resource list 
might be critiqued by one Panel/Subpanel member rather than the full complement, using a 
version of the criteria derived from the Submission Guidelines, excluding effectiveness, or other 
criteria typically used when such material is being considered for, say, publication, or inclusion 
in a video festival; a summary of these critiques could be included with the dissemination packet. 
Alternatively, the list could be presented simply as a working bibliography, along with some type 
of disclaimer. Despite the potential difficulty, it is important to share the list of information 
resources with consumers, particularly in relatively new fields, like gender equity and disability, 
where it could serve as an important educative and catalytic tool. 

What should be the role of the Subpanel Chair and what ways should be used to arrive at 

agreement? 

In my view, the roles of Chair would include advocate/spokesperson, resource person/ advisor, 
coordinator, and facilitator/mediator, with these roles often overlapping. In particular, the Chair 
would take the lead in spreading the word about the extent, nature and importance of the work 
of the Subpanel to potential Subpanel members/advisors, potential submitters, potential 
consumers, policymakers and other relevant audiences, and would encourage other Subpanel 
members to engage in similar efforts, so that the diverse perspectives and voices represented on 
the Subpanel gain visibility. 

Also, the Chair would serve as resource and advisor to all Subpanel members and to the Advisory 
Board as appropriate, ensuring that the roles and responsibilities of a member and an advisor are 
clear to everyone; identifying, collecting and disseminating seminal research, reports, etc. in the 
field so that Subpanel members begin with a common core of knowledge in addition to their 
particular expertise; laying the groundwork for the discussion and consensus-building that will 
need to be engaged in by the Subpanel regarding review criteria (relevant during the early life of 
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the Subpanel) and/or clarifying/summarizing review criteria and how to apply them to all 
members so that there is a common imderstanding and application (relevant during later stages 
of the Subpanel); and advising and serving as a resource to members on all aspects of Subpanel 
work as needed. 

Another aspect of the Chair’s responsibilities would be to coordinate the work of the Subpanel, 
ensuring the equitable distribution of submissions for review and other tasks among Subpanel 
members, engaging in follow-up with members, and ensuring that tasks and commitments of 
individual members and the Subpanel as a whole are completed in a timely manner. 

In addition, the Chair would serve as facilitator during Subpanel meetings and other collective 
work, setting in motion a climate and process that will enable members to reach decisions on core 
tasks, such as establishing review criteria and making recommendations for designation of 
submissions, and to achieve other meeting goals. Part of the facilitation process would include 
mediating disagreements on the designation of submissions. Ideally, this would involve helping 
the Subpanel evolve a workable strategy for handling disagreements that all members could 
accept. As a prerequisite to developing such a strategy, the Subpanel would need to decide upon 
their decision making process for promising and exemplary designations. For example, does there 
need to be a consensus, or will a majority vote serve? (This might be an issue to discuss across 
Subpanels and Panels to determine whether there is need for consistency. ) The type of decision 
required will affect the significance of a disagreement and whether and how it should be handled. 
To enable the Subpanel to develop a strategy for managing disagreement, the Chair might present 
a range of possibilities that have been used by other Subpanels, by grant review teams, and by 
other groups working on related tasks. Another aspect of the Chairs role as mediator/facilitator 
would be to create an environment that encourages and is respectful of different opinions, that 
allow members to change their point of view, and that facilitates but does not require 
compromise. 

In summary, the Subpanel Chair facilitates and guides, but does not dictate the work of the Subpanel. 

All Subpanel members, including the Chair, are peers, with equal authority. 
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The Gender Equity Expert Panel: The Core Gender 
Equity Sub-Panel, One Year Later 

Marylin A. Hulme, Co-Chair, Core Gender Equity Sub-Panel 



The Gender Equity Expert Panel was established by the US Department of Education, Office for 
Educational Research and Improvement in the Spring of 1996, as part of OERI’s response to 
legislation which required creating a procedure for locating and evaluating best practices, in policies, 
programs, projects, products and practices, for recommendation to the Secretary of Education. Two 
pilot expert panels were created: one on the subject of gender equity and the other on mathematics 
and science education. 

This is not the first time that attempts have been made to recommend best practices in different 
curricular areas. The recently abolished National Diffusion Network (NDN) was supported with 
substantial fimding from the US Department of Education, not only to identify those projects and 
programs that could be considered promising, but also to provide training and technics assistance 
to educators wishing to comprehend clearly and implement successfully those programs in their 
schools. The Eisenhower Consortia, in conjunction with the Regional Laboratories, have published 
a directory of promising and exemplary programs in mathematics and science education, and the 
Eisenhower National Clearinghouse is about to release a CD-ROM with recommended projects and 
information on equity in mathematics, science and technology.' 

Two of the distinguishing characteristics of these new pilot expert panels is that their membership 
includes to a great degree practitioners working in the field, and that their approach to recognizing 
exemplary and promising programs^ (and even proto-promising programs) is through the peer 
identification and evaluation process. So far, both expert panels have followed very different routes. 
Gender equity covers the entire spectrum of the educational community, from the way in which the 
school board makes policy to how teachers interact with girls and boys in the classroom, providing 
the Gender Equity Expert Panel with a very wide range of subjects from which to select and 
recommend programs. The Gender Equity Ejqjert Panel’s philosophy of working in a collegial, non- 
hierarchical marmer with participants who are very active and knowledgeable in the field of gender 
equity has led to a focus on interactions between panel members, submitters and support staff that 
are generating inquiries into what works, what are the successful programs in gender equity, leading 
to questions that have rarely been asked before, such as: 

• What is it that makes for a successful gender equity program that will have the type of 
impact that changes biased behavior? 



' Promising Practices in Mathematics & Science Education: A Collection of Promising 
Educational Programs & Practices, 1994. 

2 The word “programs” will be used to include projects, practices and products. 
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• Which programs can be seen to have made substantial institutional changes? 

• Would we find programs that could influence and inculcate “best practices” throughout 
the K- 12 system? 

• Would we find programs that not only promoted gender equity, but also provided sound 
pedagogical practice and accurate curricular content? 

• Would it be possible to assess products and practices in the same way as a fully fledged 
training program? 

• Would developers be motivated to take the time to complete the submission forms? 

• Which programs would lead to improved academic achievement for underserved, 
underrepresented students, including females? 

• Which programs would positively influence nontraditional career choice and preparation? 

Some of these questions have been answered, but others have yet to be resolved. 

This report firom the Core Gender Equity Sub-Panel provides a window on the activities of one 
particular sub-panel, which devised its operating system with assistance firom the WEEA Resource 
Center, and with little input initially firom the other sub-panels or with much attention given to 
establishing a uniform method of organizing the work. Consistency between the working habits of 
the sub-panels was not an issue; the real issue was how to get the work done with minimal fiscal 
support firom the US Department of Education. 

The Core Gender Equity Sub-Panel has two co-chairs: Marylin A. Hulme, Senior Project Associate, 
Equity Assistance Center for Region B, New York University and Rutgers University, and Dr. 
Melissa Keyes, President of Keyes Consulting, formerly the Equity Coordinator for the Wisconsin 
Department of Public Instruction. Not only does this arrangement help us to divide up and share the 
work (which is being done on a voluntary basis), but also gives us greater opportunities to generate 
ideas and diminish the impact of working in physical isolation, without face-to-face-meetings, 
however well connected we are by phone, fax and e-mail. Having a co-chair makes it easier for us 
to organize the work of our sub-panel, though we realize that this may not be true for every sub- 
panel. 

Reaching out to the gender equity community with the support of the WEEA Resource Center staff, 
who are themselves expert in the field and have created a very useful network, is helping to publicize 
the project. The sub-panel chairs collaborate in cooperatively assessing the re-allocation of 



submissions to the appropriate sub-panel when necessary. ^ When there were disagreements, this 
sub-panel’s chairs have made collaborative decisions with OERI and WEEA staff that smoothed 
progress towards the goal of the Gender Equity Expert Panel: making recommendations of gender 
equity programs that work successfully to the Secretary of Education. 

The Core Gender Equity Sub-Panel covers the following topics: 

General Instruction 
Differential Treatment of Students 
Single-Sex Schooling 

Curriculum, except for Mathematics, Science and Technology; this includes History, 
Social Studies, English, Language Arts, Literature, amongst others 
Extra-curricular Activities 
Multicultural Education 

Gender and Cultural Diversity, which includes race, ethnicity, sexual orientation. 

Linguistic Diversity 

Guidance and Counseling 

Sexual Harassment, K-12 

Physical Education and Athletics 

and anything else not covered by the other sub-panels, including general educational policies and 
procedures. 

To date, the Core sub-panel is fortunate in that its members work with the tacit support of the public 
and/or non-profit organizations to which they belong (support, being time away from the 
organization’s primary mission and duties which may be translated into considerable in-kind support 
for the persoimel involved). Composed of field-oriented people, the sub-panel sometimes finds 
applying criteria and standards derived from the educational research community and promulgated 
by OERI to a very practical, project-oriented community somewhat difficult to match, especially 
when assessing the “Evidence of Effectiveness”. This is making access to the system problematic 
for some practitioners and directors of funded projects who do not necessarily consider themselves 
researchers or evaluators, with the result that we are receiving requests for technical assistance in 
order to comply with and fulfill the objectives of the Guidelines. 

In responding to these conditions, we consider our primary tasks to be to: 

• solicit materials for submission to our sub-panel through the gender equity networks 
existing at present: the WEEA network, desegregation assistance centers, the Perkins 
grantees, and others. 



^ There are six sub-panels on these topics: Core Gender Equity, Mathematics, Science and Technology, 
Teacher Education, Vocational Technical Education, Disability, and Violence and Sexual Harassment. 



• organize the review of submitted materials through a pool of reviewers and then, 
after assessing the reviewer’s conclusions for consistency and accuracy, make 
recommendations to the entire Gender Equity Expert Panel. 

• offer some feedback through support staff to submitters/developers for further 
improvement on their submissions, especially if the material reaches the promising 
or the “proto-promising” category. 

To begin this process, we 

• brainstormed a list of programs that we would initially invite to submit 
(this list is by no means exhaustive), 

• wrote a cover letter to accompany the “request to submit” with the Guidelines, issued 
by the support contractor, in order to give the submitters a sense of who we are and 
some context of the project, 

• created a list of additional reviewers who were familiar with gender equity, but not 
officially on the Core sub-panel, 

• wrote to members of the sub-panel urging them to contribute to both lists of projects 
and reviewers , and to encourage developers to submit their programs “on their own”, 
not necessarily waiting to be invited, and, 

• warned sub-panel members that we would be calling on them to be reviewers, and 
requested their feedback on the whole expert panel process. 

Early on, we decided that all submissions be sent directly to the WEEA Resource Center, 
where each submission was acknowledged and the sub-panel chairs alerted to assign 
reviewers. We selected reviewers who we knew to have limited involvement with the 
program submitted, and who represented a measure of cultural diversity and male participants 
in our pool. If there were a question about the completeness of the submission, the WEEA 
staff and chairs conferred on the best way to request and receive the additional information. 
If the submitter’s request to have the Core sub-panel review their program seemed 
inappropriate, other sub-panel chairs were consulted and the submission transferred. 

Every program submitted to the Core sub-panel has two reviewers, and is sent out from the WEEA 
Resource Center with a rolling return date of one month and a cover letter from the sub-panel’s co- 
chairs. At present, we have about a dozen items out on review for this first pilot year. As co-chairs, 
we see our responsibilities to be looking over the primary results of the reviewers to ensure 
consistency, and to ascertain that the process is complete, and, if necessary, referring back to the 
reviewers for clarification. Before presenting a written summary of our recommendations to the 
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entire Gender Equity Expert Panel, we intend to confer with the other sub-panel chairs before the 

Panel meeting in Washington, DC, scheduled for September, 1997. 

Through our experience of applying this process, five areas of concern have come to our attention: 

• Many developers, especially those who are practitioners not researchers, are finding that the 
Guidelines can be intimidating, the process onerous and time-consuming (especially when 
the submitters are typically spread thin in their regular jobs). Data fi:om the project 
evaluations to be used for “Evidence of Effectiveness” in some cases are not sufficient to 
satisfy the criteria established by OERI; this creates difficulties and extra work from those 
submitting projects. 

Our target audience includes many practitioners who have been successful in winning 
competitive funds to develop and implement their projects. Unfortimately, “Evidence of 
Effectiveness” in their own site and also in others was not required by the US Department 
of Education at that time. Moreover, although directors of funded projects in the past requested 
assistance from the US Department of Education program officers precisely to create meaningful 
evaluations and evidence of effectiveness, the Department at that time appeared to be 
disinterested in long term evaluation and did not respond in a helpful maimer.'’ This is 
unfortunate, as directors of a considerable number of funded projects that have value and are 
known, on an informal, anecdotal basis, to have a positive impact on their target audiences 
are finding it very difficult to provide proven evaluation data. The Gender Equity Expert Panel 
as a whole must find a way to satisfy OERI standards in order to make this effort of finding and 
validating best practices in gender equity accepted and, at the same time, make the Guidelines 
more user fiiendly, so as not to fiighten off precisely those we wish to include into this process.^ 

• As a result, many submitters are requesting technical assistance of us, either to improve or 

complete a submission, in order to be able to meet the standards. We have deferred these requests 
to Dr. Joseph Maxwell, staff at WEEA, who has been assisting the submitters as far ashislime 

and resources allow. So far, he has not been overwhelmed with requests from our subpand, 

but if the number of requests increases dramatically, or if we wish to provide a agnifican t 

and effective system of technical assistance, we will have to look for alternative sources 

of technical assistance providers. It may well be possible that some of the educational researchers 
presently part of the larger “Advisory Group” of this project can be involved as technical 

assistance providers, well away fi:om the roles of the reviewers or of the sub-panel chairs so 
as not to provoke comments of conflict of interest. 



^ Personal communication from former project directors. 

* In the past, it was very common for many gender equity funded projects to create a rudimentary 
evaluation along the lines of - was the project completed, and was the target audience those intended in the 
proposal? Funding for long-term impact evaluation was practically non-existent. 
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• This leads us into another area of concern: the potential for conflict of interest. In those 
disciplines that are well established with a larger community of “experts”, conflict of interest can 
more easily be avoided. Gender equity in public education, a growing but still small field, has 
less than 1,000 recognized members, judging by membership in the national association, the 
National Coalition for Sex Equity in Education and the American Educational Research 
Association (AERA) Special Interest Group, Research on Women in Education. Complaints 
have, in fact, already been made about “fnends promoting fnends’ work”. Several well-known 
practitioners in the field have refused to submit their programs, particularly where print materials 
and media are concerned. 

What follows is a personal statement based on discussions that we (Hulme and Keyes) 
have had and continue to have. 

As the co-chairs of the Core Gender Equity Sub-Panel, we made a deliberate decision to 
remove ourselves from the initial round of reviews and also from providing technical assistance 
directly to the submitters. We have both worked in the field of gender equity for many years, 
and have close personal and professional relationships with many of the people who would be 
natural submitters. This project is based on the “peer review” process by professional colleagues. 
Because gender equity is still a fairly small discipline, there could be many “incestuous” reviews 
which might become a breeding ground for complaints and accusations of favoritism. To avoid 
this, and to be seen as avoiding this potentially negative situation, which might greatly 
detract from the success of the project, we decided that our role would be to act as initiators and 
facilitators of the process, using our own extensive knowledge of and long familiarity with the 
field of gender equity. As previously noted, requests for assistance were referred to Dr. Maxwell. 
This issue should be revisited and thoroughly discussed with the chairs of the other sub-panels, 
to ensure that a certain common level of objectivity is maintained throughout this project. 

• Cooperation with other sub-panels became important when we found that some of our 
solicitations should go directly to other sub-panels, especially in the field of mathematics, science 
and technology. We also found to our surprise that many submitters were requesting reviews by 
the Core as well as by the content area sub-panels. We accepted these requests. At present, we 
have sufficient people on our supplementary list to be able to assign reviewers without 
overloading any one person. We will continue to assign two reviewers to each Core submission, 
in order to complement any assignments by other content area sub-panels. 

• Incentives for those submitting tlieir work to the panel are lacking. Lack of time and of Interest 
in the panel’s process were cited as detriments by program developers.® The publicity of perhaps 
being included in a list of best practices for gender equity was irrelevant and not an issueftat, 
when balanced against the demanding submission process, was considered as a reward. Directors 
and product developers of programs with well-established, high reputations, that are deemed 
successful when measured by sales and/or high level of enrollment in the program, see no reason 



* Personal communication. 



to participate in this project. Yet it is precisely these programs that need to be enticed tosiimit 
and that the gender equity discipline must include in order to have a meaningful list of best 
practices. 

The following are recommended as future considerations to be discussed by the entire Gender Equity 

Expert Panel: 

• Create more outreach, both formally and informally, to pertinent organizations to involve them 
in the process by helping to identify and obtain submissions as well as to publicize this project. 
Examples are National Association for Multicultural Education, National Coalition for Sex 
Equity in Education, the network of Desegregation Assistance Centers, National Alliance of 
Partners in Equity (vocational equity network), the American Association of University Women, 
and the American Educational Research Association, Special Interest Group, Research on Women 
in Education. In addition, contact state education departments (SEAs) that have developed 
reputable nationally recognized programs but no longer have resources to support their 
dissemination and implementation . This is particularly critical due to the loss of Title IV funding 
at the state level and the incipient loss of the Sex Equity set-asides of the Carl Perkins Applied 
Technology and Vocational Education Act funds. 

• Identify from amongst the Panel’s extensive Advisory Group those educational researchers who 
are willing to provide technical assistance to submitters on a regular basis. 

• Create incentives for developers to enter this project, one of which could be funding to gather 
more substantial impact data for the “Evidence of Effectiveness”. Others could be high level 
publicity with the Department and the Secretary of Education and some system of awards. 

• Eliminate the fear that some potential submitters may have of failing to reach either category of 
“promising” and “exemplary” because of lack of formal evaluation data, by providing assistance 
to further develop their program to provide the necessary and relevant data on evidence of 
effectiveness. 

• Identify gaps in gender equity research and development that would become candidates for future 
funding, development and evaluation. 

• Create a differential submission process for products and practices, as opposed to that for 
programs and projects. Both print materials and media must be evaluated somewhat differently, 
using a qualitative review process akin to a book or film review, as the identity of the end-user 
carmot be established to glean knowledge of how the materials were used, unless extensive 
surveys were completed. 

• Provide the Gender Equity Expert Panel with fiscal support from the US Department of 
Education, OERI. We have spent this year working on a voluntary basis with the approval of our 
employing organizations to ease out the kinks of this expert panel process in a discipline that has 
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no obvious mechanisms of support. This year can be considered a pilot year; now that we can 
see the promises and the pitfalls in identifying, evaluating and recommending a gender equity 
best practices list, the Panel must be placed on firmer footing. Given the collegial strength of this 
Panel and its ability to work under less than productive circumstances, it could become an 
extremely successful model given additional help from the Department to facilitate face-to-face 
meetings, to provide consistent support for the WEEA Resource Center, and to facilitate 
responses to the queries and doubts of potential submitters. 

The activities of the Gender Equity Expert Panel have the potential to establish gender equity 
research and development as a recognized discipline, as well as setting standards of success and 
evaluation of implementation for a fairly new (25 years) field of study. We have not yet answered 
all the questions that were posed at the beginning of this paper. Neither have we discovered how 
to encourage submission of those programs that have proved their success where it is really 
significant: in the school where gender equity can assure that girls and boys have the access to the 
education that they need, with the treatment that they deserve. There can be no excellence without 
equity. 

With national efforts such as Goals 2000 and Improving America’s Schools Act (I AS A) looking for 
programs that can help engage and include all students, especially those groups of students that have 
historically been underrepresented and underserved, the development of a useful list of quality 
gender equity programs is timely and critical. 

How appropriate it is that this effort is occurring during 1997, the 25th anniversary of Title IX of 
the Education Amendments of 1972! 
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Advice on Planning a System of Expert Panels 

With Thoughts Especially on the Gender Equity Expert Panel 
by Mary M. Wiberg 

Vocational Education and School-to-Work 
Background Regarding Vocational Education Funding 

My thanks to the U.S. Department of Education and the Office of Educational Research and 
Improvement for the opportunity to serve as a member of the Gender Equity Expert Panel and for 
the invitation to provide some recommendations regarding the Expert Panel System. My 
recommendations are made fi'om the perspective of a state government employee, and focus for the 
most part on the practical and pragmatic. 

As the Gender Equity Administrator for Vocational Education in the State of Iowa, I have worked 
with schools, colleges, organizations, and my colleagues in other states in an effort to eliminate 
gender bias in vocational education. Too many of us remember days when girls could not enroll in 
“shop” classes, for they were only for boys. 

Federal vocational education legislation in the twenty-five years since the passage of Title IX has 
provided both personnel and fimding to address gender equity issues. Under the current Carl D. 
Perkins Vocational and Applied Technology Act of 1990, each state must use a minimum of 
$60,000 in federal fimds for a full-time gender equity administrator. In addition, 1 0.5% of the total 
basic Perkins grant to each state is reserved for programs to eliminate bias in vocational education 
and for programs serving single parents and displaced homemakers. 

Across the coxintry, many programs and practices have been developed as a result of these reserves 
that have increased participation of female students in nontraditional career preparation, addressed 
the needs of teen parents so they could complete their education, prepared single parents and 
displaced homemakers for high skilled jobs, and positively impacted the manner in which vocational 
education in the United States is delivered. 

While no one is certain how the next federal vocational education bill will look in final form, almost 
everyone is sure that new legislation will not continue provisions for either the gender equity 
administrator or the reserves for equity. The establishment of the Gender Equity Expert Panel is most 
timely, given the likely changes in the federal vocational education legislation, for there is still time 
to capture effective models for promoting equity in vocational education before the funding ends. 

Background Regarding the School-to-Work Act 

Included in the School-to-work Act of 1994 is extensive language requiring states to involve Gender 
Equity Administrators in the planning and implementation of School-to-Work systems, develop 
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goals to serve girls, develop strategies to provide nontraditional opportunities to female students, 
minorities and individuals vvith disabilities, and to serve “all students.” Unlike the Perkins Act, 
which is administered according to regulations from the U.S. Department of Education, a deliberate 
decision was made not to develop regulations for the STW Act, in the interest of moving system 
development forward in as speedy a manner as possible. 

The result of this decision is that minimal attention has been given to the issue of serving “all 
students,” including focusing attention on the needs of female students. In addition, a recent survey 
of state gender equity administrators reflects minimal involvement by them in the planning and 
implementation of STW grants in their states. 

I share this background information because it increases the need for and importance of the work of 
the Gender Equity Expert Panel. Identifying and capturing effective strategies that work in 
vocational education, core education, math and science, etc., will provide models to be shared with 
the newly developing STW systems at the state and local level. 

Recommendation 1: Once exemplary and promising models are identified, dissemination strategies 
should be developed which target various publics, including implementers of STW initiatives, with 
suggestions regarding how particular models can be used in a variety of settings. Where strategies 
can be translated to serving “all students,” ideas should be proposed. This may mean developing 
an additional process for the development of related ideas. 



Leadership/Management of the Expert Panel System 

During the course of the past year, those of us involved in the Gender Equity Expert panel have 
struggled to determine what made the most sense in carrying out the process of identifying 
exemplary and promising programs. We have been fortunate in the alliance between OERI and the 
Women’s Educational Equity Act Resource Center. Sue Klein has worked with us individually and 
as a group, coaching us in our sub-panel work, seeking our input into the process design. As the 
work of the Gender Equity Expert Panel moves forward, the support that has been provided will be 
needed even more. 

From my perspective, of great importance to the continued work of the Gender Equity Panel will be 
continued leadership at the U.S. Department of Education from individuals who both understand the 
myriad of gender equity issues and who are willing and able to make connections both inside and 
outside of government to help move the evaluation process forward. I commend Sue Klein for her 
untiring work in support of gender equity — always done in a quiet and professional manner. 
Katherine Hanson and Joe Maxwell have likewise provided patient support to those of us in the field. 
Too often in equity initiatives we have seen quality programs deteriorate or go out of existence when 
strong advocates are no longer involved in management and leadership. Their leadership continues 
to be needed. 



Recommendation 2: In the long-term planning for the work of the Gender Equity Expert Panel, 
provide continuity in its leadership. 

Funding for the Expert Panel System 

The discussions by members of the Gender Equity Expert Panel have been wide ranging, focusing 
on many issues. But a key issue which must be resolved relates to the identification of funding to 
support the work of the Panel. As has been noted often, while those of us involved in the Panel and 
Sub-Panels are all advocates for gender equity and other equity issues, it is possible to co mmi t only 
limited time to work on this agenda without ftmding to support the work. 

When members of the Panel or Sub-Panels have been able to meet in person, the dialogue and 
productivity are greater. Those of us who do not come fi'om a research background leam fi'om those 
who do. In like maimer, the reverse is also true there is room for the practical too. 

Though two of the sub-panels have received funding fi'om offices within the U.S. Department of 
Education, it does not appear that funding is as readily available fi'om other parts of the agency. 
While one might conclude that the uneven funding means some issues are more important than 
others, I do not believe this is true nor the intent of the Department. 

Recommendation 3: Funds must be identified to support the work of the Panel and each of the Sub- 
Panels. Where funding is less available within the agency, other sources should be sought. Because 
the work of the panel is intended to be of even quality, credible, and of value to the broad field of 
education as well as workforce development, etc., it would seem to me that the Gender Equity Expert 
Panel would be an appropriate recipient of foundation funding. Further discussions should be held 
with the Panel and with potential funders regarding this. 



Submission Process 

In the discussions with state gender equity administrators and with local program providers, one 
burning issue has been the process for submission. While all are in agreement that identifying and 
recognizing quality exemplary and promising programs is important, several barriers arise: 

• Time and intimidating guidelines . All of us are short on time. The guidelines 
themselves require a sizable block of time to complete. Because the intent of 
identifying quality requires documented evaluation, the very language is foreign 
to many successful program operators/developers. In fact, for those who are 
unfamiliar with research studies, the language is intimidating. 

Recommendation 4; Consider the submission guidelines for exemplary programs used by the 
National Center on Research in Vocational Education as a model for a slicker-looking document. 
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That does not mean less rigor in the evaluation process. (This is a surface issue, but something 
glossy may result in greater response.) 

Lack of national evaluation data 

While millions of dollars have been spent on gender equity in vocational education in the past twelve 
years, the quantity of good evaluation regarding programming success is very limited. Hindsight 
is great, and many of us wish that federal legislation had mandated greater accountability for 
measuring the impact of funds. But that was not a clear requirement of the Perkins Act, and often 
evaluation efforts counted processes and people rather than results. The absence of evaluation data 
puts program developers/operators in the frustrating position of not being able to submit initiatives 
they may know are effective. (I should add that all gender equity coordinators at the state and local 
levels have grown much more aware of the need for data which shows the impact of programs.) 

Recommendation 5: In an effort to assist implementers in conducting quality measurement of 
program impact, develop some simple and straight-forward guidelines about designing evaluation. 
OERI and the Gender Equity Panel could work together on a piece for the non-researcher. It could 
be distributed through various organizations, educational groups, etc., and could lead to successful 
submissions at a later date. In addition, the guidelines could serve to assist individuals in planning 
for evaluation and collection of impact data in the design of equity programs. 

Need for more evaluation data 

Another issue vrith regard to evaluation data or the lack of it is that programs often do not have 
fimding to support third party evaluation. While third party evaluation is certainly not the only way 
to demonstrate the effectiveness of programs, most evaluation incurs financial or personnel cost. 

Recommendation 6: OERI should fund evaluation for promising programs through an application 
process (perhaps competitive) in order to more rapidly identify exemplary programs. 

Dissemination 

Regardless of how many exemplary and promising programs are identified, unless the dissemination 
process is well-thought out and reaches educators and other service providers who might replicate 
the programs, the Expert Panel system will not have accomplished its intent. Publication of findings 
in professional journals is very unlikely to reach the program providers who need information 
regarding exemplary programs and strategies that work. 

Recommendation 7a: The U.S. Department of Education should expand its web page to include 
information about exemplary programs. However, in order to engage the viewer, various sorts 
regarding who the program serves, what the focus is, cost, and impact should be defined. Where 
possible, web pages should be hot linked to other pages which give more information regarding 
programs. 
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Recommendation 7b: Update information on exemplary and promising programs on a regular 
b^is. If a promising program has received new evaluation which designates it as exemplary, 
highlight such information on the web page. 

Recommendation 7c: Develop print information which can be downloaded from the web page in 
order to provide additional information about specific programs. Formatting of such documents 
should be such that it v^ill be attractive to copy and disseminate at the local level. 

Recommendation 7d: Develop strong relationships with the national School-to-Work Office as a 
mechanism to disseminate information. Direct contact can also be made with all state STW offices. 
Within the Department of Education itself, other offices such as those supporting vocational 
education, special education, etc., should be engaged in the dissemination process. 

Review process 

A great deal of important discussion has taken place regarding the review process. If the 
recommendations of the Expert Panels are to be valid and credible, it is essential that there be 
consistency in the review process. 

Recommendation 8: Avoid any conflict of interest with regard to which reviewers review which 
submissions. Assure that all submissions receive equal treatment to assure a fair review process. 

Common Understanding of Evaluation Data 

As noted above, many program providers assume that because their evaluation process is not 
research-based, they do not have adequate data to demonstrate effectiveness. Submitters and 
reviewers alike should be operating from a common understanding regarding evaluation data. The 
following questions/comments are examples of considerations around evaluation data. 

1 . Do not assume that all data should be quantitative. 

2. People from outside the program can provide anecdotal data which support the promising or 
exemplary nature of the program. 

3. How integrated is any program into the system it serves? 

4. How well do people perform in the workplace as a result of participation in the program? 

5. Look at the cost per unit and assess its realistic use of fimds. 

Recommendation 9: OERI and WEEA should develop some guidance above and beyond the 
evaluation criteria to serve as a base for common understanding among the evaluators. 

Diversity in the Membership of Subpanels and Reviewers 

In my conversations with others who work in the area of equity, when I have named the subpanels 
on gender equity I have been asked, “Isn’t there a panel regarding minority girls and women?” My 
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response has been that our intent is that each of the panels will be aware of the unique needs of 
various groups and highlight what submissions have demonstrated effectiveness in serving those 
groups. I would not propose that a separate group be established specific to the issue of color, 
though the existence of the panel on disabilities provides an argument coimter to that. However, I 
have real concern that the membership of the subpanels and the reviewers be reflective of the 
diversity in our culture. Too often we pay attention to black and white groups, but overlook Asians, 
Latinos, etc. 

Recommendation 10: OERI and WEEA should review the current memberships and reviewers for 
purposes of identifying gaps in representation. Where there is not adequate representation of 
diversity, specific requests for nominations should be issued. Not all qualified members and 
reviewers must come from the educational equity arena. For example, the National Association of 
Commissions for Women might be able to identify strong advocates for women who also represent 
diverse groups. 

Conclusion 

The opportunity to write this paper provided me with time to think about the work of the Subpanel 
on STW and Vocational Education. I look forward to learning the recommendations of the other 
chairs and to implementing those appropriate as we move forward in this important enterprise. 



Lessons to be Learned from Setting Up the Gender 
Equity Expert Panel for Establishing 
a System of Expert Panels in Education 

by Patricia E. Ortman, Ph.D., Chair 

Teacher Education and Professional Development 
Gender Equity Expert Panel 

The following are my recormnendations, informed by my personal experiences as a participant in 
the process of orgamzing the Gender Equity Expert Panel and my understanding of the ongoing 
conversations among all of the developers of this panel, for developing the system of expert panels 
in education. I have concentrated on questions of process because I believe that we want to build 
a system of expert panels which operate successfully for a long time. In order to do so, we must 
provide a solid foundation. My recommendations are made to accomplish the goal of providing each 
panel, and thus the system of panels, with that solid foundation for the conduct of their work. None 
of my recommendations is meant, nor should be construed, as criticisms of the processes utilized to 
create the Gender Equity Expert Panel, as we are blazing a trail and I believe that we've done the best 
we could and are continuing to do the best we can under the present constraints. However, I also 
believe that we can learn from the process to inform the creation of the future panels and the main 
point of my recommendations is, in fact, that the department modify the conditions of creation in 
order to allow development of the panels to proceed in a more orderly, reasonable fashion. As a 
taxpayer, I want my government to be frugal, but as a citizen I know that my best interests are served 
by a well-educated population, which can only be accomplished if educators are aware of and trained 
in the best programs and practices and have knowledge of and access to the best products. 

Specific recommendations: 

• Determine ahead of time criteria for specifying panelists in each area. I recormnend that, 
in addition to considerations of ethnic diversity, each panel be represented by equal 
numbers of educators in each area, researchers in the field, members of appropropriate 
non-govermnental professional organizations, and government people. Additionally, each 
of the potential panelists should have as an area of expertise the area the panel is set up 
to address. Although the teacher education subpanel has terrific ethnic diversity (one 
African-American female, one Hispanic American female, one Asian American male, one 
Euro- American female), we are all college professors. There is no one on the panel from 
an appropriate professional organization such as the American Association of Colleges 
of Teacher Education, for instance; nor is there a person who has done original research 
on gender equity, per se. Of the four approved panelists, three have done research 
focused much more on multi-cultural education than gender equity and one is not even 
particularly interested in that area at the present time. 

• In a systematic fashion, identify people who qualify (meet the specific criteria believed 
to be important for that particular panel), contact them ahead of time, give them full 



information about the nature of the panel and its work, the criteria for invitation, and the 
requirements/responsibilities (see # 3 below) of being a panelist. Verify interest, 
qualifications, and availability. (In any communications, use simple, clear, 
straightforward language. Especially for practitioners not used to reading government 
documents, government language can be dense and obfuscate rather than clarify points.) 
I attended my (the?) first meeting of the Gender Equity Expert Panel because the general 
written invitation sparked my interest and curiosity but I really went to see what it was 
all about, rather than to get down to work. Actually, I believe that that meeting may in 
fact have been an informational rather than an organizational meeting, but my belief is 
that its mission wasn't entirely clear to everyone since we ended up doing what I 
consider to be organizational tasks (e.g., creating subpanels). At any rate, potential 
panelists need to be identified and then contacted by phone to discuss the panel and their 
level of interest and qualifications, etc., and then, if they are interested, available, 
qualified, they should be issued an official invitation to participate in the first 
organizational meeting. I recommend that you invite only those people you anticipate 
appointing, who agree to attend and to serve if appointed. 

Along with providing initial invitees clear and sufficient information about the nature and 
work of that particular panel, provide potential panelists with a clearly defined and 
concise definition of their responsibilties. When I agreed to serve on the panel itself, I 
believed that the extent of my involvement would be advisory (e.g., providing ideas about 
how to obtain submissions and the like) and reviewing submissions on an occasional 
basis. When I agreed to serve as chair of the subpanel, I thought that I would be 
coordinating submission reviews. I didn't realize that I would be involved with actually 
setting up the guidelines for review as well as the submission and review processes 
themselves; nor did I realize that I would be the person chiefly responsible for obtaining 
submissions. Part of my personal ignorance on this matter was a function of not being 
at the second (first?) organizational meeting and points up the importance of my next 
recommendation. 

Pay the expenses of invited participants attending the first and all subsequent meetings, 
as well as a per diem honorarium. I missed what I consider in retrospect to be a (the?) 
crucial (second?) organizational meeting of the Gender Equity Panel because of an 
invitational oversight, but even if I had been invited, I would not have been able to attend 
unless traveling expenses had been subsidized. Others may not be able to participate 
without both expenses and a per diem, at least not on a regular basis. The panels' success 
seems to me to be a function of panelists being able to participate on a regular basis. And 
if the work of the panelists does involve setting up the panels' procedures and guidelines 
in addition to a submission and review system as well as reviewing submissions on more 
than an occasional basis, this is much more work than most people can volunteer on an 
on-going basis. Therefore, it appears imperative to me that each panel will need some 
kind of minimal operating budget for panel members' expenses. Additionally, money 
should be budgeted for commissioned research syntheses (see recommendation # 7). 
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Convene the first meeting only when an almost whole panel exists with committed people 
who would be willing to serve if appointed. (Do not appoint anyone who does not attend 
at least two of three organizational meetings.) Convene AT LEAST three organizational 
meetings. Have as an early order of business the filling out of the panel by having the 
panel assess its own needs in terms of additional areas of expertise and identifying other 
panelists who may meet them. Again, although the teacher education subpanelists are all 
qualified to be on the panel in some or many ways, only one of us had ever been involved 
in ANY of the original organizational meetings and that was only one of what I believe 
to have been three. Even the level of information the other three panelists had about the 
panel before their official appointment was at worst non-existent and at best minimal. 
Two subpanelists did not even know they had been appointed to the panel, let alone what 
the panel was about, until I called them to discuss subpanel work. 

The full panel should meet together AT LEAST twice to get started and analyze the 
nature of the work before dividing the panel into subpanels, if that is what they decide to 
do. In the case of the Gender Equity Expert Panel, divisions of content/labor were 
determined at the first (orgamzational/informational) meeting I attended and this was 
before much of anything else was anywhere near clear. In retrospect, I am not entirely 
certain that we would have subdivided along the same lines had we put more thought into 
the nature of our work and the process(es) ahead of time. Perhaps the nature of and 
problems with accessing information about the state of gender equity education in teacher 
education programs would be an issue which the whole panel would want to have thought 
about and addressed in a more systematic fashion rather than through creating another 
subpanel with which few wanted to be involved and about which no one present appeared 
to know the "state of the art." Perhaps teacher education and professional development 
would/should not even have been dealt with as a separate subpanel; perhaps it 
would/should have been an area which the panel as a whole decided to investigate in 
order to articulate the gaps in that domain. 

At the initial orgamzational (and, in fact, all subsequent) meetings, the panel should 
assess the state of education in that domain at that point in time, like the National 
Institutes of Health OMAR panels referred to by Lois-ellen Datta in her discussion with 
Michael Scriven. She refers to the OMAR panels as the "gold-standard for carefully 
assessing evidence of treatment effectiveness" and points out that they begin every review 
with a "first-class, extensive, independent" review of the literature, a research synthesis. 
This brings the panelists up to date so that they are all starting fi-om the same point, the 
point of knowledge of the most recent events in their fields. I agree with Lois-ellen that 
in case a recent review of the type the panelists need in order to do their best work is not 
available, they should have the power, money and time to commission such a review 
synthesis. Money to do this needs to be built into the panel's budgets. 

In the case of the first organizational meetings of the panels, after having benefit of the 
first such review of the research, panelists should analyze the specific nature of the work 
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of that panel and decide the most effective way to divide the work and the most effective 
way to provide leadership of the panel's activities. Both the structure of the panel and its 
leadership should be determined by the panelists themselves (and if they decide to divide 
into subpanels, each subpanel should decide how leadership on that subpanel will work). 
Being chair of the teacher education subpanel came to me more by default than by an 
active decision on the part of any of the subpanelists, including me. Although I am 
willing and able, I do not think that is the model that should be used for future panels. 

Each panel should then determine reasonable goals and set a reasonable pace. I am not 
sure that it has been particularly productive to establish procedures and processes at the 
SAME time as soliciting submissions and beginning the reviews. I think a better strategy 
may be to work as a whole panel to set up procedures and then do one or at most two 
submissions as pilots to begin to iron out the major wrinkles. At the present time we 
seem to have a lot of submissions, but it seems that they may be a bit stalled in the 
system; we are backed up in the review process while still having many, many 
unanswered questions. That's not to say that we won't continue to have questions arise 
as processes get more and more smoothed out, but the whole experience has been perhaps 
a little more chaotic than may be necessary. It might be better if panels created a list of 
tasks, a logical order, and a reasonable timeline. I think it is often better to go more 
slowly and establish a firm foundation than to go too fast and maybe crash. It is to be 
hoped that the other panels will not be functioning under the kind of severe time 
constraints that this Gender Equity Expert Panel seemed to have to be created under, 
however. 

Therefore, in line with the previous recommendation I specifically recommend to each 
succeeding panel that they establish submission and review guidelines and processes 
fairly firmly before they solicit submissions. Additionally, the panelists should identify 
potential problems inherent in generating submissions in each area they look at. They 
should try to anticipate questions of submitters by asking themselves how they would 
respond to a call for submissions under the conditions they consider implementing and 
then try to answer them or change the conditions of submission. The key question, 
"What's in it for submitters?" must be one that is high on each panel's agenda. As Joe 
Maxwell pointed out in his talk at AERA, obtaining submissions of programs has been 
more difficult than anticipated, due to a number of factors; (perceived) difficulty of the 
submission process, a (perceived) lack of incentives to submit a program, and lack of 
awareness of the importance of personal contact in persuading developers to submit their 
programs for review. I also think that for teacher education and professional development 
programs there are problems in supplying the kind(s) of evidence which it is perceived 
the DOE regulations value and require. As Melissa Keyes points out in her memo of 21 
April 1997 via e-mail to Joe Maxwell, as someone who received money from the Federal 
government to do gender equity work, she knows the difficulties in finding "time and 
money for evaluation" and "effective and acceptable methods of evaluation" of those 
programs she developed. She also points out that even the Department of Education was 



not able to help her after repeated requests. This would seem to indicate that the 
Department of Education may be holding developers to standards of effectiveness that it 
does not know how to measure, or if it does, it is apparently not telling. This seems just 
a little unfair. At any rate, most practitioners do not gather anything more than traditional 
evaluative evidence, which it seems DOE may hold to be weak evidence, so it may be the 
case that other panels also need to grapple with the "evidence of effectiveness" question 
as well. 



The dissemination question is also one which I recommend panels undertake to answer 
early, as it also helps to answer the question of "Whafs in it for submitters?" If 
information is actually going to be efficiently disseminated to practitioners with 
recommendations to use their programs, this could or should be highly motivating to 
program developers. 

Once the submission/review/dissemination process questions are fairly settled, each panel 
will need to design a comprehensive program to get submissions. It is not efficient to 
have the submission process be so dependent upon personal intervention by panelists. 
One idea may be a pre-submission submission, similar to the kind of thing I originally 
sent out over the Women's Studies listserv and in flyers at a few conferences. Others 
have also suggested this kind of thing, i.e. send us your good ideas and your phone 
number, address, and we will get back to you. In line with that, panelists should have the 
ability to provide guidance and money for "testing" of programs that do not yet have the 
kinds of evidence DOE requires in order to obtain that kind of evidence. 

Additionally, each panel should have the benefit of one full-time staff person to do the 
administrative work. That person should also be responsible for obtaining most of the 
submissions. It is clear fi-om our experience on the Gender Equity Expert Panel that one 
person who devotes less than half time to the work of the panel is just not enough, unless 
we want to continue to progress at a relatively slow rate (or slower) rate, which should 
also be considered a viable option in the case of continued limited support from DOE. 



Provisions must be made for regular meetings of both the full panel and any subpanels. 
There must be at least an annual meeting and semiannual meetings would be preferable. 
Along with that, the establishment of a panel listserv for each panel is a necessity. This 
has worked well for the Gender Equity Panel, in my opinion. 

Last, but not least, it is my strong recommendation that ALL panels must consider 
questions of gender equity in their deliberations. For instance, one of the next proposal 
panels is in the area of technology. There is growing evidence that girls are discouraged 
from computer use at very young ages. How teacher educators are trained to intervene 
in this process is important for girls’ futures. It would be unconscionable for a panel of 



educational experts in technology not to address those and related issues. Each panel 
should identify the gender equity and teacher education/professional development issues 
it needs to address in its field and attend to them in appropriate ways. 
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